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*^The  study  began  with  a review  of  past  SPT  efforts  to  determine  what  was  done  and  to  evaluate  the  strengths  and  weak- 
nesses of  each  approach.  The  troubleshooting  task  was  then  analysed  to  determine  discrete  behavioral  steps  and  the  informa- 
tion requirements  associated  with  each  step  The  SPT  concept  was  developed  around  the  results  of  these  analyses.  } 

With  the  SPT  concepts  firmed  up,  the  presentation  mechanisms  (i.e.,  visuals  and  data  tables)  were  designed;  This  design 
dictated  the  parameters  of  SPT  materials  to  be  produced  As  earlier  SPT  researchers  have  observed,  providing  complete 
operating  information  in  symbolic  form  can  fill  volumes  and  require  extensive  production  time.  Equipment  materials  were 
produced  once  for  normal  equipment  operation  Problem-specific  information  was  produced  only,>ofihe  subset  of  materials 
affected  by  an  individual  item.  Clerical  production  was  accomplished  similarly  usrng  magnetic  Storage  for  normal  data  and 
inputting  only  the  problem-specific  changes. 

The  SPT  materials  were  validated  prior  »o  the  full-scale  data  collection.  The  validation  revealed  a major  problem  with  the 
answering  scheme  and  several  minor  problems  with  the  visuals  Validation  personnel  commented  on  the  difficulty  associated 
with  initially  learning  the  SPT  concept  and  materials  use  The  overall  concept  appeared  workable  and  the  individual  results 
matched  expectations.  The  answering  scheme  end  the  visuals  were  modified.  As  a result  of  the  validation,  a practice  problem 
was  also  developed  to  provide  free  practice  prior  to  a subject's  symbolic  performance  testing. 

The  field  testing  was  conducted  at  Lowry  Air  Force  Base,  Colorado,  from  mid-January  to  mid-February  1978.  Fifteen 
students  and  16  administrators  were  tested  in  groups  of  four  for  two  days  each.  One  test  administrator  monitored  two  sets 
of  concurrent  Jot)  Sample  Tests  (JSTs)  and  SPTs  Data  collected  included  answer  (suspected  malfunctioning  stage),  time  to 
completion  steps  to  completion,  and  a record  of  check  sequence  and  location. 

/Analysis  of  the  results  indicated  similar  performance  on  both  JST  and  SPT  forms.  The  accuracy  scores  for  all  subjects  on 
all  tests  produced  a positive  correlation  of  384  which  is  significant  at  the  .025  level. 

Time  to  completion  produced  a positive  correlation  of  .588  which  was  significant  at  the  .0005  level.  Steps  to  completion 
produced  a positive  correlation  of  .356  which  was  significant  at  the  .025  level 

Analysis  of  the  check  sequences  and  locations  produced  very  high  positive  correlations  between  performance  on  JST  and 
SPT  forms 

The  primary  difficulty  encountered  was  highly  variable  troubleshooting  performance  regardless  of  test  form.  This  vari- 
ability is  illustrated  by  the  fact  that  the  subjects  on  thelaverage  missed  two  problems  out  of  every  three.  In  this  regard, 
results  were  similar  to  those  of  earlier  studies. 

v^The  results  indicate  that  the  SPT  approach  was  sound,  requiring  very  few  modifications.  Several  new  applications  are 
Suggested  by  the  results;  these  include: 

f 1 .N)  Adapting  the  method  for  computer  presentatiory 

( 7.  Using  the  method  to  provide  practice  in  training,  in  addition  to  testing^  ^ 

Using  some  of  the  SPT  equipment  analyses  for  maintenance  evaluation  and  in  curriculum  development/  t'! 

4 ) Applying  the  method  to  SPTs  tor  other  levels  of  troubleshooting  penetration. 
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INTRODUCTION 


Increased  emphasis  on  performance-oriented  training  requires  the 
development  of  more  job-relevant,  valid,  and  performance-oriented  ap- 
proaches to  skill  measurement  than  the  multiple-choice  questions  now 
in  vogue  in  Air  Force  maintenance  courses.  One  obvious  alternative 
approach  would  be  to  give  actual  job  sample  tests  using  actual  equip- 
ment. Such  job  sample  or  job  task  performance  tests  can  provide  high 
levels  of  content  and  face  validity.  However,  these  te.c  often  re- 
quire prohibitive  amounts  of  test  administrator  time  £ may  require 
scarce,  fragile,  and  expensive  prime  equipment  and  tes  equipment. 

In  addition,  such  tests  are  sometimes  hazardous  for  both  students  and 
equipment.  For  these  reasons,  job-oriented  testing  on  operating 
equipment  has  not  been  used  or,  at  best,  has  been  used  in  such  a 
limited  way  that  test  results  have  been  neither  valid  nor  reliable. 
Resources  are  not  now  available  to  increase  on-equipment  testing  nor 
are  such  resources  likely  to  be  available  in  the  future.  A reasonable 
solution  may  be  found  using  a paper-and-pencil  or  computer-based  simu- 
lation approach,  which  reduces  test  administrator  burdens  and  does  not 
require  testees  to  perform  on  tactically  configured  hardware  and  to  use 
scarce  tools  and  test  equipment.  Simulated  performance  tests  are 
strongly  cor, unended  by  AFM  50-2  as  an  approach  for  operating  within 
training  resource  and  safety  constraints. 


Background 


Foley  (1974)  prepared  an  extensive  literature  review  concerning 
the  evaluation  of  maintenance  performance.  He  cited  several  studies 
which  showed,  without  exception,  low  correlations  between  job  task  per- 
formance tests  and  conventional  pencil-and-paper  theory  and  knowledge 
tests.  Similar  results  were  found  by  Engel  and  Rehder  (1970)  who  con- 
cluded, for  the  specific  Military  Occupational  Specialty  examined,  that 
the  methods  of  evaluation  in  use  (i.e.,  multiple-choice,  job  knowledge 
tests)  did  not  possess  an  acceptable  level  of  validity  within  a group 
or  by  individual  measurement  to  support  the  existing  personnel  system 
requirements.  Their  conclusion  could  probably  be  generalized  to  any 
career  area  using  conventional  pencil-and-paper  theory  and  knowledge 
tests  for  evaluation. 

Bergman  and  Siegel  (1972)  reviewed  the  literature  concerning  train- 
ing evaluation  and  student  achievement  measurement.  They  concluded  that 

(1)  there  was  too  much  use  of  rational  rather  than  empirical  methods; 

(2)  there  was  too  much  subjectivity  when  objectivity  was  needed;  and 


( 
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(3)  evaluation  research  was  too  often  limited  by  monetary  considerations. 
They  also  stated  that  most  of  the  research  studies  reviewed  were  techni- 
cally deficient  because  of  (1)  t he  use  of  too  few  subjects;  (2)  the  use 
of  inappropriate  statistical  techniques;  (3)  the  failure  to  use  control 
groups,  or  use  of  inadequate  controls;  ( A ) the  use  of  improper  sampling 
procedures;  or  (3)  the  use  of  inappropriate,  contaminated  or  unreliable 
criteria. 

Requirements  for  criterion-referenced  job  task  performance  tests 
were  strongly  emphasized  by  Foley  (1974).  Kngel  (1970)  commended  the 
work  sample  criterion  as  a reliable  and  job-relevant  measure  that  also 
could  be  used  as  a standard  for  the  evaluation  of  other  measurement  tech- 
niques. Advantages  of  job  sample  (job  task  performance  tests)  cited  by 
Siegel,  Bergman,  Federman  and  Sellman  (1972)  were  (1)  realism;  (2)  prac- 
ticality; (3)  objectivity;  (4)  content  validity;  ami  (3)  freedom  from 
verbal  requirements.  Guidance  in  the  preparation  of  these  performance 
tests  (at  varying  levels  of  useful”)  s;)  is  contained  in  Boyd  and  Shimberg 
(1971)  and  Vineberg,  Taylor,  Young,  ilirshfeld  and  Maier  (1976).  High 
cost  and  difficulty  of  administration  m job  task  performance  tests  are 
widely  noted  (Engel  & Rehder,  1970;  Foley,  1974;  Foley,  1975;  Osborn, 

1970;  Siegel  et  al.,  1972).  Osborn  (1970),  using  the  term  "-ynthetic 
performance  tests,"  recommended  selection  of  inexpensive  alternatives 
to  fully  job-relevant  performance  tests.  Cost  trade-offs  should  be  made. 
Foley  (1974,  1975)  quite  explicitly  demanued  empirical  validity,  stating 
that  job  task  performance  tests  should  be  used  in  spite  of  their  high 
cost,  if  they  are  the  only  empirically  valid  tests  availabr0. 

Fitzpatrick  and  Morrison  (1971)  cited  two  symbolic  performance  tests 
as  tests  of  diagnostic  problem-solving  performance:  the  Tab  Test  for 
troubleshooting  (Glaser,  Damrin,  & Gardner,  1952),  and  the  medical  simu- 
lation exercises  of  McGuire  and  Babbott  (1967). 

Foley  (1974)  discussed  empirical  validity  of  early  pencil-and-paper 
substitutes  for  job  task  performance  tests.  Of  particular  note  were  the 
symbolic  equipment  tests  as  represented  by  the  Tab  Test  (Crowder,  Morri- 
son & Demaree,  1954)  and  the  Multiple-Alternative  Symbolic  Troubleshoot- 
ing Test  (MAST)  (Grings,  Rigney,  Bond  & Summers,  1953).  These  tests  dis- 
played equipment  schematic  diagrams.  Tabs  were  used  to  cover  displays 
of  information  normally  obtained  by  using  test  equipment  on  the  actual 
hardware.  Corks  were  substituted  for  tabs  in  the  MAST.  Correlations 
with  job  task  performance  tests  were  found  to  be  minimal  in  studies  by 
Crowder  et  al.  (1954)  and  Evans  and  Smith  (1953). 

Foley  (1974)  expressed  the  strong  opinion  that  a different  approach 
to  the  development  of  symbolic  performance  tests  could  result  in  higher 
correlations.  However,  the  results  in  an  initial  attempt  to  provide 
better  symbolic  performances  in  the  area  of  troubleshooting  showed  a neg- 
ative correlation  with  job  task  performance  tests  (Shriver  & Foley,  1974). 
Initial  validation  of  symbolic  troubleshooting  was  deemed  unsuccessful. 
Following  modification,  Shriver  and  Foley  (1974)  then  found  symbolic  tests 
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tii  have  a hig.i  degree  ot  empirical  validity  (when  using  job  task  perfor- 
mance tests)  at  the  chassis  or  black  box  level  (although  not  when  higher 
numbers  ot  alternatives  were  possible,  as  at  the  component  or  piece-part 
level ) . 


Several  considerations  and  hypotheses  were  presented  to  influence 
future  development  ot  symbolic  performance  tests.  Of  particular  note 
were  the  suspected  inability  of  many  subjects  to  use  test  equipment  prop- 
erly, the  distractions  and  interruptions  typically  found  on  the  job,  and 
the  need  for  inclusion  of  a more  realistic  presentation  of  information 
(Including  the  use  of  random-access  projection  of  test  equipment  readings) 
In  support  of  the  final  consideration,  Lefkowith  (1955)  showed  realism  to 
be  an  important  (actor  in  the  validity  of  pictorial  tests  (although  there 
was  a practical  limit). 


Written  simulation,  a technique  simulating  the  decision-making  pro- 
cesses of  doctors  and  others  involved  in  diagnosing  and  managing  patient 
problems,  lias  been  widely  used  in  the  health  professions  for  testing  and 
certification.  McGuire,  Solomon,  and  Bashook  (197b)  suggested  that  the 
essence  of  simulation  could  he  captured  in  a pencil-r  1-paper  format  em- 
ploying either  latent  image  or  opaque  overlay  techniques  for  feedback 
systems.  Tlus,  written  simulations  have  been  claimed  to  provide  economic 
and  technical  feasibility  for  self-assessment  and  large-scale  testing  in 
varied  settings.  McGuire  et  al.  (1976)  asserted  that  the  methodology  of 
written  simulation  is  widely  applicable  to  an  alm-st  unlimited  variety  of 
content  areas,  educational  levels,  and  management  settings.  The  tech- 
nique, however,  has  not  been  successfully  applied  to  troubleshooting  in 
an  Air  Force  environment.  Similar  techniques,  however,  have  been  used 
for  training  (Cantor  & Brown,  1956;  Naval  Training  Device  Center,  1960). 


Objectives 


The  aim  of  this  effort  was  to  develop  and  evaluate  the  practical 
usefulness  of  a paper-atid-pencil  simulation  approach  to  performance  test- 
ing in  a technical  training  environment.  It  was  expected  that  the  ap- 
proach could  be  generalized  to  a broad  range  of  testing  situations.  The 
focus  of  the  effort  was  on  corrective  maintenance  of  electronic  equipment- 
a pervasive  maintenance  problem  common  to  a wide  variety  of  Air  Force 
Specialty  Codes  (AFSCs) . More  specifically,  the  effort  focused  on  the 
measurement  of  electronic  troubleshooting  skill — widely  seen  as  the  most 
critical  skill  underlying  overall  electronic  maintenance  performance  and 
a skill  which  has  been  studied  extensively  by  behavioral  scientists  for 
over  three  decades. 
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Troubleshooting 


Troubleshooting  is  v( iwed  within  this  study  as  a systematic  process 
for  gathering  information  about  external  and  Internal  malfunction  symp- 
toms, leading  through  the  functional  dependency  relations  (data  flow) 
among  equipment  components  to  a deduced  conclusion  as  to  the  component 
causing  the  externally  observed  symptom.  The  process  is  characterized 
by  dependency  relation  analysis  (information  gathering  using  test  equip- 
ment) and  sequential  decision  making  (i.e.,  deciding  what  information  to 
collect  and  ultimately  what  component  to  adjust  or  replace). 

The  activities  inherent  to  the  process  are  described  below,  in  the 
usual  order. 


Activity 


1.  Symptom  Detection 


2.  Symptom  Pattern  Completion 


3.  Symptom  Pattern 
Analysis 


4.  Output  Deficiency 
Analysis 


5.  Specific  Experience 


Process 

The  failure  of  the  equipment  to  per- 
form to  specification  is  noted.  Ini- 
tiating cues  include  write-up,  sche- 
duled check,  or  operational  failure. 

When  equipment  malfunctions,  some  num- 
ber of  equipment  outputs  will  be  af- 
fected. The  logic  of  troubleshooting 
first  checks  which  outputs  are  bad  and 
then  looks  for  a cause  which  is  common 
to  all  the  bad  outputs.  Much  internal 
checking  can  be  avoided  by  noting  which 
outputs  are  good  and  bad. 

With  a complete  symptom  pattern,  stages 
not  associated  with  bad  outputs  can  be 
eliminated  without  internal  checking. 

In  some  cases,  the  character  of  the 
symptom  itself,  rather  than  its  rela- 
tion to  other  symptoms,  focuses  on 
particular  groups  of  stages  as  the 
only  ones  which  can  cause  that  sort 
of  symptom. 

(A  special  case  of  Activity  4.)  This 
sort  of  knowledge  is  usually  associated 
with  long  experience  with  the  equip- 
ment during  which  the  technician  learns 
to  associate  probable  causes  with  spe- 
cific output  conditions. 


Summary  of  Approach 


Briefly,  the  approach  was  to  develop  an  equipment-free,  paper-based 
simulation  of  the  hardware  which  the  testee  could  troubleshoot  in  the 
same  logical  manner  as  he  would  troubleshoot  the  real  hardware,  thus 
requiring  similar  knowledges  and  cognitive  skills.  In  this  case,  the 
hardware  consisted  of  the  HP  652A  oscillator  and  the  Tektronix  453A 
oscilloscope,  commonly  used  by  Air  Force  technicians.  It  was  expected 
that  troubleshooting  task  performance  on  the  simulation  would  be  pre- 
dictive of  performance  of  the  same  tasks  on  the  actual  hardware.  To 
test  this  assumption,  two  groups  of  subjects — journeyman  technicians 
represented  by  instructors  in  the  Precision  Measuring  Equipment  (PME) 
course  at  Lowry  Air  Force  Base,  Colorado,  and  novices  represented  by 
students  in  the  same  course  (who  had  completed  training  on  this  equip- 
ment)— were  given  the  sar.o  problems  in  two  test  modes:  criterion  or 
Job  Sample  Test  (JST)  and  simulation  or  Symbolic  Performai  * Test  (SPT). 
There  were  15  students  and  16  instructors  who  were  given  tnree  problems 
on  each  piece  of  equipment  in  each  mode — 3 x 2 x 2 or  12  problems  in  all. 


Test  Development  and  Simulation  Issues 


The  Criterion  Problem 


It  is  important  to  recognize  that  criterion  validity  is  not  assured 
through  use  of  actual  job  tasks  as  test  items.  Whether  such  tasks  are 
used  or  not,  it  is  necessary  that  the  variety  of  tasks  be  sufficient  to 
account  for  the  variability  in  the  hypothetical  ultimate  criterion  and 
that  the  test  tasks  be  performed  under  conditions  similar  to  those  found 
on  the  job.  Further,  if  validity  is  to  be  assured,  the  scoring  of  indi- 
vidual items  must  reflect  their  importance  in  the  context  of  the  total 
job  or  task. 

It  is  possible  that  a performance  test  with  all  of  the  above  char- 
acteristics (which  would  make  its  contents  valid)  could  be  entirely  use- 
less because  of  unreliability.  Thorndike  (1949)  notes  that  it  is  not 
necessary  for  the  reliability  of  a criterion  measure  to  be  extremely 
high.  It  is  necessary  that  the  criterion  measure's  reliability  be  other 
than  zero,  and  that  the  reliability  be  known,  since  if  it  is  not,  vali- 
dation of  other  instruments  against  the  criterion  will  not  be  possible. 

Even  if  the  criterion  test  produces  valid  and  reliable  measures  of 
performance,  it  may  still  fail  in  usefulness  because  it  is  not  diagnos- 
tic. If  the  criterion  measures  will  be  used  to  validate  instruments 
which  will  predict  only  success  or  failure  on  the  job,  and  if  success  or 
failure  truly  expresses  the  limit  of  our  interest,  then  such  a criterion 
would  be  useful.  However,  such  a criterion  would  not  be  useful  if  we 


were  interested  in  knowing  why  those  who  failed  did  so,  what  specific 
tasks  proved  too  difficult  for  them,  or  what  training  or  performance  aids 
would  be  required  to  bring  them  to  an  acceptable  level  of  competence.  It 
is  questions  of  this  latter  sort  in  which  we  are  most  often  interested. 


Simulation 


If  all  important  behavioral  components  of  a task  are  represented  in 
the  simulation  in  about  the  same  way  they  are  represented  in  the  real 
world,  performance  in  the  two  situations  should  be  similar.  To  achieve 
good  simulation  (i.e.,  good  behavioral  verisimilitude),  the  opportunities 
for  error  or  inadequate  performance  which  would  degrade  performance  in 
the  real  world  must  be  found  to  about  the  same  extent  in  the  simulation. 
Note  that  nothing  has  been  said  about  physical  resemblance.  The  focus 
must  be  on  the  behavioral  components  of  tasks.  In  general,  money  spent 
on  simulation  for  reasons  other  than  to  create  an  opportunity  to  exhibit 
the  behaviors  critical  to  task  performance  must  be  justified  on  grounds 
other  than  fidelity  of  simulation. 

Good  simulation  must  be  based  on  careful  analysis  of  the  tasks  for 
which  simulation  is  desired.  Just  as  too  much  attention  to  physical  re- 
semblance may  not  be  cost  effective,  too  little  attention  to  behavioral 
similarity  can  lead  to  simulations  which  are  of  little  value  in  predict- 
ing real-world  performance. 

Previous  work  in  troubleshooting  task  simulation  has  been  plagued 
by  several  difficulties.  Chief  among  them  is  that  tasks  have  been  modi- 
fied for  simulation  purposes  in  such  a way  as  to  make  them  different 
(usually  easier)  than  the  tasks  performed  on  the  hardware. 

Some  previous  efforts  have  restricted  the  range  of  troubleshooting 
behavior  sampled  in  the  JST  to  what  was  possible  to  simulate  with  a given 
technique.  While  this  will  lead  to  better  prediction  of  success  on  the 
JST  from  knowledge  of  performance  on  the  SPT,  the  generality  of  the  re- 
sult is  limited  by  the  partial  JST  behavior  sample. 

Systematic  efforts  have  rarely  been  made  to  establish  the  content 
validity  of  the  JST,  and  no  study  has  been  found  which  has,  for  elec- 
tronic troubleshooting,  empirically  related  performance  on  the  JST  to 
overall  job  performance.  Further,  it  has  often  been  impossible  because 
of  resource  limitations  to  develop  sufficient  numbers  of  items  to  assure 
measurement  reliability,  either  on  the  JST  or  SPT.  Additionally,  SPTs 
have  frequently  failed  to  predict  performance  on  JSTs  because  the  behav- 
ior sample  domains  of  the  two  tests  were  different,  typically  in  that 
many  behaviors  required  on  the  JST  were  not  required  on  the  SPT.  Out- 
standing examples  of  such  omissions  are  use  of  test  equipment  and  trans- 
lation of  data-flow  representations,  such  as  schematics,  to  physical  rep- 
resentations of  the  hardware. 


12 


Troubleshooting  is  often  a higher  order  task  with  many  components. 
Skill  on  one  or  some  of  the  components  may  not  always  predict  whole-task 
skill.  Absence  of  some  critical  skill  elements  (use  of  test  equipment 
is  an  excellent  example)  can  be  completely  disqualifying,  and  skill  in 
all  components  does  not  assure  whole-task  competence. 

For  some  technicians,  "making  it  all  play  together"  is  a significant 
problem.  Keeping  track  of  where  one  is  in  the  deductive  process — what  is 
known  and  wiiat  still  must  be  found  out — while  continually  being  inter- 
rupted with  interpolated  tasks,  such  as  getting  access  to  test  points 
and  setting  up  test  equipment,  adds  an  important  dimension  to  task  diffi- 
culty. 

Three  other  important  difficulties  with  troubleshooting  simulations 
follow: 

1.  Error  cues  have  been  provided  in  the  SPT  which  would  not 
be  present  in  the  JST.  This  has  resulted  often  from  the 
difficulty  of  providing  equal  error  opportunity  in  the 
SPT.  To  do  so  implies  providing  information  on  test 
points  not  ordinarily  considered  relevant  to  the  prob- 
lem being  solved.  Information  on  non-relevant  test 
points  has  frequently  presented  cues  to  the  testee 
indicating  that  he  was  on  a non-relevant  test  point 

and  cuing  him  to  alter  his  strategy. 

2.  The  SPT  by  its  very  nature  fosters  the  use  of  a dif- 
ferent approach  to  troubleshooting  by  the  technician 
in  that  all  tests  tend  to  be  of  equal  effort,  whereas 

in  the  JST  there  is  often  wide  variance  in  the  effort  re- 
quired to  make  a test.  The  tendency  of  technicians 
is  to  maximize  information  gained  per  unit  check  cost 
where  the  cost  of  the  check  is  in  terms  of  the  diffi- 
culty of  gaining  access  to  the  test  point,  waiting  for 
the  system  to  cycle  into  the  right  state,  hooking  up 
additional  test  equipment,  using  test  equipment  with 
which  he  is  unfamiliar,  etc.  This  means  that  he  tends 
to  make  cheaper  rather  than  more  expensive  checks  when 
troubleshooting  real  hardware.  Check  costs  tend  to  be 
equal  in  the  SPT  mode,  and  thus  check  cost  is  not  a 
variable. 

3.  Simulations  have  tended  to  assume  that  the  system  was 
always  in  the  appropriate  state  making  it  unnecessary 
for  the  testee  to  be  concerned  about  system  state  and, 
indeed,  considerably  reducing  his  opportunity  for  error 
(system  state  equals  a unique  set  of  positions  for 
switches,  including  internally  controlled  switches). 
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The  following  section  covers  how  the  JST  (criterion)  and  SPT  (sim- 
ulation) were  developed  and  how  the  testing  was  conducted  in  this  effort. 
Results  and  conclusions  will  be  found  following  the  methods  discussion. 
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METHODS 


Criterion  Development 


Criterion  development  took  the  general  form  of  identifying  potential 
malfunction  problems  in  each  piece  of  equipment  and  then  assessing  the 
representativeness  of  each  possible  problem  as  well  as  assessing  the  prac- 
tical difficulty  in  isolating  the  cause  of  a malfunction. 

During  the  pre-item  selection  and  equipment  analysis,  the  equipment 
was  studied  and  the  schematic  diagrams  were  partitioned  at  the  stage  lev- 
el. The  relative  difficulty  of  locating  any  given  stage  as  the  cause  of 
a malfunction  was  assessed.  The  difficulty  factor  was  then  used  to  group 
possible  items  into  three  classes  ranging  from  hard  to  easy  to  locate 
problems . 

Item  selection  and  validation  were  based  on  the  expertise  of  PMEL 
instructors  with  field  experience  on  the  equipment  to  be  used  in  the  test. 
Their  selections  were  made  after  an  analysis  of  the  equipment  and  exam- 
ination of  the  assigned  difficulty  ranking. 

Once  a consensus  was  reached  on  which  items  to  use,  failed  compo- 
nents within  the  stage  and  failure  modes  (e.g.,  open,  short,  change  in 
value)  were  specified.  A major  concern  was  the  design  of  malfunctions 
that  were  not  visually  detectable. 


Pre-Item  Selection/Equipment  Analysis 

The  desired  level  of  penetration  was  to  the  stage  level.  The  first 
task,  therefore,  was  to  divide  the  equipment  into  stages.  Selection  of 
stages  was  accomplished  by  looking  first  at  the  system-level  block  dia- 
gram, as  shown  in  the  vendor-supplied  maintenance  manual.  Selection  was 
made  to  correspond  as  closely  as  possible  to  the  individual  blocks  on  the 
system-level  block  diagram  in  the  maintenance  manual.  Choice  of  stages 
was  indicated  by  drawing  partitioning  lines  on  the  schematics.  Figure  1 
is  a system-level  block  diagram,  and  Figure  2 is  a schematic  of  part  of 
the  equipment  with  the  stage  lines  shown.  The  following  definition  was 
used  as  a guide  in  determining  stages: 

A stage  is  generally  considered  to  be  one  step  in  a 
multi-step  process.  An  amplifier  stage  might  be  a single 
transistor  or  integrated  circuit  and  its  associated  compo- 
nents, such  as  resistors  and  capacitors,  while  an  oscillator 
or  multivibrator  stage  may  contain  two  or  three  transistors 
and  their  associated  components. 
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Schematic  Partitioned  into  Stages 


Test  Item  Selection  Criteria  Development 


The  project  was  concerned  with  the  information-gathering  and  sequen- 
tial decision-making  elements  of  troubleshooting.  The  behaviors  of  con- 
cern were  the  following: 

1.  Deciding  where  in  the  data  flow  to  test. 

2.  Selecting  the  test  instrument  and  test  parameter. 

3.  Identifying  the  physical  test  point  in  the  hardware. 

4.  Reading  the  test  result. 

5.  Deciding  whether  the  outcome  is  good  or  bad. 

In  stage  level  troubleshooting,  this  group  of  behaviors  is  repeated 
for  the  sequence  of  points  of  test  selected  by  the  testee  until  the  stage 
containing  the  malfunction  is  identified.  At  the  behavioral  level,  there 
should  be  no  difference  between  the  JST  and  the  SPT.  The  objective  in 
developing  these  selection  criteria  was  to  characterize  the  population 
of  problems  on  the  equipment  so  that  those  of  known  difficulty  could  be 
selected. 

Previous  research  has  shown  that  troubleshooting  difficulty  is  a 
function  of: 


1.  The  complexity  of  the  data  flow,  where  complexity 

is  equal  to  the  number  of  components  plus  the  number 
of  interconnections  among  them. 

2.  The  kind  and  amount  of  test  equipment  ordinarily 
available  for  use  in  checking. 

3.  The  information  available  in  the  performance  aids 
on  readings  and  tolerances. 

4.  The  information  available  in  the  performance  aids 
and/or  placarded  on  the  hardware  concerning  test 
point  locations. 

5.  The  data  flow  presentation  itself.  (Some  data  flow 
presentations  greatly  facilitate  deciding  where  in_ 
a system  to  check  because  of  their  layout.  Indeed, 
some  presentations  are  supplemented  by  symptom  cause 
charts  or  dependency  charts  which  also  aid  in  deciding 
where  to  check. ) 

Another  criterion  to  be  considered  was  failure  frequency.  Frequency 
of  failures  in  most  electronic  equipment  is  heavily  skewed,  with  a rela- 
tively small  percentage  of  components  accounting  for  a relatively  large 
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percentage  of  failures.  This  fact  must  be  taken  into  consideration  in 
selecting  test  items  if  they  are  to  be  truly  representative  of  the  normal 
troubleshooting  fault  population. 

Equipment  Breakdown  by  Troubleshooting  Difficulty.  A matrix  was  de- 
signed to  characterize  relative  problem  difficulty.  Refer  to  Figure  3 
for  an  illustration  of  the  matrix  format.  The  rows  of  the  matrix  repre- 
sent the  stages.  The  matrix  columns  list  factors  which  affect  trouble- 
shooting difficulty,  including: 

1.  Type  of  data  flow  representation  in  the  maintenance 
manual. 

2.  A listing  of  test  equipment  required  for  troubleshooting. 

3.  Type  and  amount  of  reading  and  tolerance  information 
available  in  the  performance  aids  for  each  parameter 
appropriate  to  the  test  instrument  in  question  (e.g., 
for  a voltmeter,  readings  and  tolerances  may  be  avail- 
able for  voltages,  but  not  for  current  or  resistance; 
or  readings  and  tolerances  may  be  available  for  wave- 
forms, but  not  RMS  values  measured  on  a voltmeter. 

This  is  usually  a function  of  the  maintenance  manual 
for  a particular  piece  of  equipment). 

4.  The  number  of  stages  and  interconnections  among  them 
within  the  functional  group. 

5.  Failure  frequency. 

The  matrix  permits  identification  of  problems  representative  of  all 
required  behaviors  (such  as  scope  display  interpretation,  meter  reading, 
deciding  where  to  check  next  in  the  absence  of  normal  reading,  and  infor- 
mation in  the  performance  aid)  and  will  reflect  those  problems  most  often 
encountered  in  actual  maintenance.  The  following  paragraphs  discuss  the 
contents  of  each  matrix  column. 


Equipment  Breakdown — A listing  of  stage  designations  for  the 
equipment  being  analyzed.  The  stage  designations  consisted  of 
the  reference  designator  for  the  active  circuit  component  in  a 
stage,  such  as  Qll  for  a transistor  or  CR4  for  a diode. 

Failure  Rate — An  indication  of  high,  normal,  or  low  failure  rate 
determined  by  analyzing  logistics  failure  rate  data.  On  this 
effort,  these  entries  were  supplied  by  field  experts  during  the 
problem  validation. 


Complexity  (Number  of  Connections) — The  total  number  of  inter- 
connections among  the  components  within  a particular  stage. 


DOCUMENTATION  FEATURES  OPTIMAL  SOLUTION  CHARACTERISTICS 


Figure  3.  Equipment  Breakdown  by  Troubleshooting  Difficulty  Matrix 


Point  of  Test  Accessibility  (Hard  or  Easy) — An  assessment  of 
ease  of  access.  Accessibility  was  judged  "Easy"  if  components 
were  located  on  the  top  of  printed  circuit  boards  and  had  ref- 
erence designators  nearby  making  points  of  test  easy  to  find. 
Accessibility  was  judged  "Hard"  if  a second  layer  of  protective 
covers  had  to  be  removed  to  provide  access  to  points  of  test, 
or  if  the  stages  in  question  had  feedback  loops  connected  to 
them  which  ordinarily  would  be  broken  in  troubleshooting.  If 
the  stages  had  feedback  loops  which  were  not  available  on  con- 
nectors or  terminals  (indicating  that  a component  would  have  to 
be  desoldered  or  a foil  strip  broken  to  break  the  feedback  loop 
in  order  to  inject  synthetic  feedback),  accessibility  was  also 
rated  "Hard." 

The  "Documentation  Features"  portion  of  the  matrix  covered  the  main- 
tenance manuals  made  available  to  the  troubleshooter  on  the  test  equip- 
ment specified.  This  part  of  the  matrix  was  comprised  of  the  following 
five  categories. 

Schematic  Coverage — If  the  stage  had  schematic  diagram  coverage 
to  the  component  level,  a "Yes"  was  indicated.  If  not,  a "No" 
was  indicated. 

Symptom  Cause  Coverage — If  the  documentation  contained  a Symptom 
Cause  Chart  operating  at  the  desired  level  of  maintenance,  a "Yes" 
was  indicated.  If  not,  a "No"  was  indicated. 

Troubleshooting  Tree  Coverage — If  the  documentation  had  trouble- 
shooting trees  to  identify  malfunctioning  stages,  a "Yes"  was 
indicated.  If  not,  a "No"  was  indicated. 

Parts/Point  of  Test  Location  Coverage — T f the  maintenance  docu- 
mentation contained  information  in  which  individual  component 
locations  were  specified,  the  column  cell  entry  was  a "Yes."  If 
it  did  not,  the  cell  entry  was  a "No." 

Circuit  Description  Coverage — The  circuit  descriptions  con- 
tained in  the  documentation  were  checked  to  see  if  they  dis- 
cussed individual  stage  operation.  If  the  circuit  descrip- 
tions contained  only  superficial  mention  of  a stage  and  its 
function,  the  cell  entry  was  a "No."  If  the  stage  and/or 
its  function  were  discussed  in  sufficient  detail  (electron 
current  flow),  the  cell  entry  was  a "Yes." 

The  "Optimal  Solution  Characteristics"  were  derived  by  doing  a paper- 
and-pencil  troubleshooting  exercise  on  each  segment  of  the  data  flow  that 
could  be  isolated  as  a cause  of  a single  missing  output  from  the  device. 

A partial  data  flow  drawing  was  constructed  for  the  stages  necessary  to 
produce  the  missing  output.  Troubleshooting  of  the  data  flow  diagram 
then  consisted  of  half-splitting  the  data  flow  and  recording  the  check 
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number  at  each  point  of  test.  In  this  case,  checks  upstream  or  downstream 
from  an  earlier  check  were  given  the  same  number  at  the  next  half-split 
point  (see  Figure  4). 

For  example,  in  a data  flow  diagram  of  eight  series  blocks,  the  first 
check  would  be  between  blocks  4 and  5 and  would  be  labeled  1.  The  next 
upstream  check  would  be  between  blocks  2 and  3 and  would  be  labeled  2. 

The  next  downstream  check  would  be  between  blocks  6 and  7 and  would  also 
be  labeled  2.  Troubleshooting  of  each  data  flow  drawing  was  completely 
performed  in  this  manner. 

The  test  equipment  required  to  measure  signals  at  the  designated 
points  of  test  was  also  noted.  The  schematic  diagram  or  other  mainte- 
nance data  were  checked  to  see  if  readings,  waveforms,  or  other  relevant 
signal  parameters  were  present.  If  the  value  was  not  present,  "No  Value" 
was  indicated  adjacent  to  the  check  number.  For  components  with  more 
than  one  input,  it  would  not  matter  which  input  received  the  next  check 
number,  or  the  next  check  number  plus  one.  Figure  5 shows  an  optimal 
solution  including  this  type  of  information. 

Number  of  Troubleshooting  Steps — For  the  following  explanation, 
refer  to  Figure  5.  The  number  of  troubleshooting  steps  was 
indicated  by  the  highest  check  number  recorded  on  any  of  the 
inputs  to  the  stage,  plus  the  number  of  checks  associated  with 
the  stage,  plus  as  many  checks  as  were  required  to  check  all 
of  the  inputs  to  the  particular  data  flow  diagram.  Ir  the 
example  illustrated  by  Figure  5,  the  number  of  troubleshooting 
steps  is  seven,  including  one  for  the  power  supplies,  one  tor 
each  of  the  inputs,  and  four,  the  highest  check  number  on  an 
input. 

Number  of  Steps  No  Values--Total  number  of  "No  Value"  steps 
("No  Value"  steps  are  checks  at  points  of  test  wich  signal 
values  not  provided  by  the  maintenance  documentation).  Check 
3 or  4 (the  output  of  block  9)  on  Figure  5 is  a "No  Value"  step. 

Number  of  Input  and  Power  Supply  Steps — The  number  of  input  and 
power  supply  steps  was  the  total  of  the  power  supply  checks  and 
the  input  checks  required  to  verify  correct  power  supply  and  sig- 
nal inputs  to  the  data  flow  diagram. 

Number  of  Steps  Using  Scope — The  number  of  checks  made  with  the 
scope  was  recorded  in  this  cell.  Teats  made  on  the  power  supply 
and  the  inputs  with  the  scope  were  also  included. 

Number  of  Steps  Using  Volt-Ohm-hilliammeter  (VOM) — The  number  of 
checks  made  with  the  VOM  to  isolate  the  problem  was  entered  in 
this  cell.  Tests  made  on  the  power  supply  and  the  inputs  were 
again  included. 
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Figure  A.  Data  Flow  Drawing  with  Optimal 

Solution  Check  Numbers  Indicated 
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Figure  5.  Data  Flow  Drawing  with  Optimal  Solution 
and  Necessary  Test  Equipment  Indicated 
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Number  of  Steps  Using — There  was  space  provided  in 

the  matrix  for  several  of  these  columns.  These  columns  were 
provided  in  anticipation  of  the  later  discovery  that  additional 
test  equipment  was  required  to  troubleshoot.  Examples  of  other 
equipment  might  include  signal  generators,  power  supplies,  and 
power  meters.  The  cell  entry  would  be  the  number  of  required 
checks  made  with  the  equipment  during  the  optimal  solution. 

Problem  Difficulty — Problem  difficulty  is  a number  ranging  from 
zero  to  100.  The  higher  the  number,  the  more  difficult  the  prob- 
lem, according  to  the  algorithm.  This  number  was  computed  and 
entered  on  the  matrix  after  all  other  matrix  cells  were  filled. 

Problem  Difficulty  Algorithm.  The  problem  difficulty  algorithm  had 
two  forms:  one  with  an  accessibility  factor  and  one  without.  After  re- 
viewing a completed  matrix,  if  it  was  discovered  that  all  of  the  accessi- 
bility entries  were  the  same,  the  algorithm  without  the  accessibility  fac- 
tor could  be  used,  since  accessibility  was  then  a constant  and  would  not 
change  the  results.  Dropping  accessibility  makes  the  calculation  of  the 
individual  difficulty  numbers  easier  and  faster. 

Both  forms  of  the  algorithm  contain  weighting  factors  which  were 
estimated  by  troubleshooting  experts.  These  estimates  were  made  by 
judging  the  relative  importance  of  each  algorithm  factor  to  the  trouble- 
shooting task.  The  experts  judged  that  test  equipment  use  and  hard- 
ware operating  values  were  most  important,  of  equal  value,  and  account 
for  75  percent  of  troubleshooting  difficulty.  The  remaining  three  fac- 
tors were  judged  approximately  equal  in  importance  and  assigned  weights 
of  8.3  each  to  account  for  the  final  25  percent.  The  algorithm  weights 
of  these  were  adjusted  to  12.5  in  the  version  which  omits  accessibility. 

Both  forms  of  the  difficulty  algorithm  are  shown  in  Figure  6. 

(Refer  to  this  figure  during  the  following  discussion,  to  which  it  is 
keyed. ) 

1.  The  first  term  in  the  equation  is  a complexity  factor, 
determined  by  taking  the  number  of  stage  connections  for 
a given  stage  and  dividing  it  by  the  number  of  stage  con- 
nections in  the  largest  stage  of  the  equipment.  The  result 
is  then  multiplied  by  a constant  weighting  factor  of  8.3 

in  the  algorithm  version  including  the  accessibility  fac- 
tor, or  12.5  in  the  algorithm  version  omitting  the  acces- 
sibility factor. 

2.  The  second  term  in  the  equation  is  the  accessibility 
factor  (if  it  is  required).  Where  there  are  differences 
in  accessibility,  the  matrix  entries  are  rated  either 
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Problem  Difficulty  Algorithms  (Keyec 


"Easy"  or  "Hard."  In  the  formula,  a "Hard"  entry  equals 
a constant  of  8.3;  an  "Easy"  equals  zero.  In  the  calcula- 
tions here,  each  term  in  the  algorithm  reduces  to  a number 
which  Is  summed  with  the  numbers  from  the  other  terms  to 
provide  the  difficulty  factor;  thus,  "Hard"  accessibility 
adds  8.3  to  the  total. 

3.  The  third  factor,  divided  Into  two  parts,  accounts  for  the 
amount  of  help,  or  its  inverse,  the  lack  of  troubleshooting 
help  supplied  by  the  available  documentation  and  critical 
dec  is  ion  informat  ion. 

a.  The  first  part  deals  with  the  number  of  avail- 
able features  in  the  documentation  and  the  num- 
ber of  features  that  are  applicable  to  trouble- 
shooting a given  stage.  The  number  of  applicable 
features  for  a given  row  is  obtained  by  counting 
the  cells  with  the  "Ycsses";  this  number  is  then 
subtracted  from  the  total  number  of  features,  ob- 
tained by  counting  the  number  of  columns  under 
"Documentation  Features."  The  result  of  this 
subtraction  is  then  multiplied  by  the  result 

of  dividing  8.3  by  the  total  number  of  features. 

b.  The  second  part  consists  :'f  dividing  the  number 
of  troubleshooting  steps  with  no  values  (obtained 
from  that  column  in  the  matrix)  by  the  total  num- 
ber of  troubleshooting  steps  (obtained  from  its 
column),  and  multiplying  the  result  by  37.5. 

4.  The  fourth  factor  in  the  equation  is  a composite  of  subordi- 
nate equations,  one  for  each  piece  of  test  equipment  used 

in  the  optimal  troubleshooting  solution.  Each  factor  con- 
sists of  the  total  number  of  measurements  performed  with 
a given  piece  of  test  equipment,  multiplied  by  the  test 
equipment  difficulty  factor.  That  quantity  is  then 
divided  by  the  total  number  of  steps  in  the  optimal  solu- 
tion. This  operation  is  repeated  for  each  piece  ol  test 
equipment  using  the  same  process.  The  results  are  then 
summed  and  multiplied  by  a constant  of  12.5.  The  test 
equipment  difficulty  factors  used  on  this  project  were: 

3 for  an  oscilloscope  or  signal  generator,  2 for  a power 
supply,  and  1 for  a VOM  or  a Digital  Voltmeter  (DVM) . 


Test  I tern  Val  ldat ion  and  Selection 

The  matrices  were  validated  by  five  Air  Force  PMKL  instructors,  win' 
evaluated  information  presented  and  noted  any  significant  deviations  from 
field  practice,  it  was  possible  that  test  equipment  specified  in  the 


maintenance  manuals  was  not  typically  available  In  the  field  or  that  test 
equipment  commonly  used  In  the  field  was  not  specified.  In  addition,  the 
troubleshooting  routines  designed  to  present  optimal  solution  to  the  prob- 
lems might  have  failed  In  one  way  or  another  to  take  advantage  of  special 
procedures  used  In  the  field. 

The  validation  consisted  of  the  experts  either  agreeing  or  disagree- 
ing with  the  difficulty  assessment  of  the  potential  items.  The  test  items 
were  taken  from  the  list  of  those  candidate  problems  agreed  to  by  all  of 
the  experts.  At  the  conclusion  of  the  validation,  the  problem  character- 
ization and  the  minimum  Inventory  of  test  equipment  required  for  trouble- 
shooting were  agreed  upon. 
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One  hard,  one  average,  and  one  easy  item  were  selected  from  the  list 
ol  possible  items  for  both  pieces  of  equipment  to  be  used  in  the  test. 

At  this  point,  each  Item  was  specified  as  a malfunctioning  stage.  It  was 
then  necessary  to  select  a component  to  be  failed  and  a failure  mode.  In 
selecting  a failure  mode,  it  was  important  to  select  one  which  would  not 
cause  additional  damage  to  the  equipment.  Additional  equipment  damage 
would  confuse  the  results,  since  there  would  be  more  than  one  problem  to 
be  solved  in  the  equipment. 

The  malfunctions  used  in  the  test  to  be  discussed  here  consisted  of 
the  following: 

Tektronix  A53A  (Scope) 

t} 504  - open  emitter/base  junction  - hard  - DF*  ■ 77. A 
Q923  - open  emitter/base  Junction  - average  - DF  ■ 51.7 
Q1255  - open  emitter/base  junction  - easy  - DF  - AO. 8 

Hewlett-Packard  h52A  (Oscillator) 

A1Q9  - shorted  emit  ter /base  Junction  - hard  - DF  *■  bA.7 

A1CR19  - shorted  - easy**  - DF  - A7.8 

A3Qb  - open  emitter/base  Junction  - easy  - DF  » A8.2 

Malfunction  Installation  techniques  were  considered  from  the  stand- 
point of  ease  of  Installation  and  removal,  and  also  from  the  standpoint 
of  providing  visual  clues  to  the  troubleshooter  which  might  make  malfunc- 
tion Isolation  easier. 


*DF  - Difficulty  Rating 

**This  item  was  originally  selected  as  average,  but  upon  reexamination 
was  Judged  easy. 
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Test  Development 


Job  Sample  Test 

The  JST  was  designed  to  be  a re-creation  of  the  normal  troubleshoot- 
ing task  and  environment.  Actual  malfunctioning  equipment  was  used.  Nor- 
mal test  equipment  and  maintenance  manuals  were  available  for  use  in 
troubleshooting.  The  test  administrator  checked  the  equipment  operation 
between  items,  installed  the  malfunctions,  and  gave  the  testee  a terse 
write-up  of  the  problem.  The  testee  was  free  to  pursue  any  strategy  in 
locating  the  malfunction  short  of  wholesale  parts  replacement. 

The  only  additional  or  test-specific  requirement  was  for  the  testee 
to  fill  out  a protocol  form  during  the  troubleshooting.  The  testee  noted 
the  point  of  test  and  type  of  instrument  used  for  each  check  made  during 
fault  isolation. 


Symbolic  Test 

Previous  research  has  not  provided  a highly  reliable  predictive  sym- 
bolic substitute  test  for  electronic  troubleshooting.  Some  tests  were 
conceived  as  part-task  performance  analogues. 

In  his  Annual  Review  of  Psychology  article  on  "problem  solving  and 
thinking"  in  1959,  Gagne  said,  "To  summarize,  troubleshooting  of  complex 
equipment  typically  consists  of  problem  solving  which  is  sequential  in 
nature;  there  is  a sequence  of  hypotheses  that  must  be  tested  in  order 
to  narrow  progressively  the  area  in  which  the  malfunction  is  located." 
Earlier  research  developing  and  testing  symbolic  substitute  tests  dealt 
only  with  the  abstract  logic  of  problem  solving  in  troubleshooting,  such 
as  the  studies  by  Crowder,  Morrison,  and  Demaree  (1954)  and  by  Evans  and 
Smith  (1953),  each  of  which  produced  low  correlations. 

Shriver  and  Foley  (1974)  suggest  that  none  of  the  earlier  tests  "in- 
cluded any  of  the  'distractions'  from  the  main  line  of  'problem  solving' 
found  in  the  real  world  of  troubleshooting.  In  the  job  environment  an 
individual  must,  for  example,  set  up  and  operate  his  test  equipment  to 
obtain  test  point  information,  as  well  as  to  obtain  instructions  and  in- 
formation from  his  Technical  Orders."  These  distractions  interrupt  the 
analytic  problem-solving  thought  process  and  contribute  to  the  overall 
task  difficulty.  They  also  increase  the  opportunity  for  procedural  and 
interpretive  errors  which  may  inappropriately  modify  the  problem-solving 
strategy.  It  is  very  possible  that  troubleshooting  difficulty  is  greater 
than  the  sum  of  the  individual  component  behavior  difficulties. 
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In  a discussion  of  their  revised  symbolic  troubleshooting  test  de- 
sign, Shriver  and  Foley  (1974)  used: 

1.  Pictorial  test  equipment  displays  (meter  faces  and 
oscilloscope  displays). 

2.  Procedural  simulation  (subject  was  not  required  to 
follow  any  particular  strategy  by  test  materials). 

3.  Integrated  testing  (test  situation  and  materials 
required  subject  to  interact  with  the  usual  mate- 
rials and  equipment — perform  the  normal  behaviors 
symbolically — and  to  integrate  the  results  to  con- 
clude the  nature  of  the  malfunction). 

4.  A component  replacement  option  with  associated  dis- 
play changes  for  replacing  a defective  component. 

5.  A test  administrator  for  every  subject. 

The  symbolic  testing  reported  by  Shriver  and  Foley  (1974)  covered 
fault  isolation  and  three  levels  of  penetration: 

1.  Major  unit  (black  box) 

2.  Individual  circuit  (stage) 

3.  Component  (piece-part) 

At  the  major  unit  level,  symbolic  results  matched  criterion  perfor- 
mance 87  percent  of  the  time.  At  the  individual  circuit  level,  results 
matched  67  percent  of  the  time,  and  results  matched  53  percent  of  the 
time  at  the  component  level. 

Shriver  and  Foley  (1974)  concluded  that  as  the  actual  troubleshoot- 
ing task  required  use  of  more  test  equipment  to  isolate  malfunctions, 
instances  of  matching  performance  on  symbolic  and  criterion  tests  dropped. 
They  suggested  that  symbolic  substitute  tests  for  test  equipment  operation 
be  used  to  screen  potential  subjects.  While  this  approach  would  certainly 
produce  a positive  effect  on  the  evaluative  statistics,  it  does  not  deal 
directly  with  the  issue  of  making  the  symbolic  test  more  equivalent  to 
the  actual  job,  and  thus  more  predictive  of  actual  performance  on  the 
job. 

It  seems  reasonable  that  a symbolic  test  permitting  the  exercise  of 
all  useful  troubleshooting  behaviors  would  be  highly  correlated  with  on- 
the-job  performance. 
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Analysis  of  troubleshooting  behavior  suggested  that  the  following 
subtasks  be  addressed  by  the  symbolic  test  design: 

1.  Manipulating  the  equipment  state  (front  panel  control 
configuration)  to  discern  the  character  or  probable 
location  of  the  malfunction. 

2.  Obtaining  operating  parameters  from  any  point  of 
test  within  the  equipment  (where  point  of  test  is 
any  junction  of  two  or  more  components). 

3.  Choosing  any  of  the  test  instruments  normally  avail- 
able for  troubleshooting  to  measure  static  or  dynamic 
operating  parameters  (via  the  troubleshooter's  pre- 
ferred approach) . 

4.  Physically  locating  the  desired  point  of  test. 

5.  Specifying  the  desired  test  parameter. 

6.  Setting  up  and  connecting  the  test  instrument  to 
obtain  the  desired  measurement. 

7.  Reading  displays  and  factoring  range  and  function 
information  to  obtain  measurement  values. 

8.  Judging  the  test  results  (value  is  in  or  out  of 
tolerance) . 

9.  Repeating  the  above  sequence  until  the  faulty  com- 
ponent can  be  specified. 


The  design  described  by  Shriver  and  Foley  (1974)  covered  three  lev- 
els of  troubleshooting,  and  the  materials  varied  somewhat  from  level  to 
level.  When  viewed  separately,  the  stage  level  isolation  test  did  not 
contain  items  1,  2,  4,  6,  or  7.  Our  hypothesis  is  that  this  test  was 
less  typical  of  the  job  than  the  black  box  level  test  and  accounts  for 
most  of  the  excess  variation  between  criterion  and  symbolic  results  in 
the  stage  level  comparison. 


Task  analysis,  item  selection,  validation,  test  equipment  identi- 
fication, and  malfunction  design  combined  to  supply  the  information 
handling  requirements  of  the  SPT.  The  information  to  be  provided  by 
the  SPT  included  the  following: 


1. 


2. 


3. 


Access  to  all  possible  electrical  points  of  test. 
Access  to  front  panel  controls  to  change  system  state. 
Access  to  front  panel  displays. 
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4.  Access  to  test  equipment  range  and  function  controls. 

5.  Test  equipment  displays  for  all  combinations  of  range, 
function,  system  state,  and  points  of  test. 

The  goal  of  the  format  design  was  to  provide  the  easiest  possible 
access  to  the  information  listed  above  in  a cost-effective  way.  The  num- 
ber of  possible  equipment  and  test  equipment  displays  was  very  large. 

The  HP  b 52A,  the  smaller  of  the  two  pieces  of  equipment  used,  contained 
a minimum  of  280  possible  points  of  test.  (Many  points  of  test  have  sev- 
eral locations;  in  situations  like  this,  only  one  was  counted.)  If  one 
assumes  six  system  states,  six  range  and  function  settings  for  an  oscil- 
loscope used  for  testing,  and  24  range  and  function  choices  for  the  volt- 
meter to  be  used  in  testing,  then  there  are  241,920  possible  test  dis- 
plays. Judicious  selection  of  problems  and  system  states  reduced  this 
number  to  41  waveforms,  25  voltmeter  faces,  and  six  front  panel  displays. 

The  large  reduction  in  total  displays  required  was  achieved  by  choos- 
ing system  states  for  the  oscillator  which  were  frequency  or  amplitude 
multiples  of  each  other.  This  permitted  repeated  use  of  the  same  wave- 
form display  in  association  with  different  settings  of  the  oscilloscope 
used  for  troubleshooting.  The  VOM  displays  were  disassociated  with  the 
range  and  function  choices  so  that  25  displays  covered  all  possible 
meter  displays  (one  display  for  every  other  minor  scale  division  on  the 
DC  scale).  This  practice  provided  a voltmeter  display  within  the  combined 
tolerance  of  the  prime  equipment  and  the  VOM  for  any  possible  value. 

The  sequential  matrix  format  was  selected  as  the  mechanism  for  fix- 
ing the  relationships  of  the  variables  to  arrive  at  an  output  display. 

The  matrix  sequence  followed  the  usual  diagnostic  protocol  of: 

1.  Selecting  a system  state  and  point  of  test. 

2.  Deciding  on  a diagnostic  test  and  specifying  the  test 
equipment  required. 

3.  Setting  up  test  equipment  (selecting  range  and  function). 

4.  Connecting  test  equipment  to  observe  test  display. 

5.  Analyzing  results,  repeating  sequence,  or  making  indicated 
repair. 

The  matrix  sequence  was  begun  by  physically  selecting  a point  of 
test  from  equipment  photographs  with  points  of  test  numbered.  System 
states  were  selected  from  equipment  line  drawings  depicting  the  various 
combinations  of  front  panel  control  settings.  Desired  test  equipment  was 
selected  by  going  to  the  proper  matrix.  Test  equipment  range  and  func- 
tion selections  were  made  from  line  art  drawings  depicting  the  various 
function  and  range  combinations. 
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The  SPT  was  designed  to  provide  the  troubleshooter  with  both  the 
same  information  requirements  and  the  same  Information  sources  found  in 
the  real-life  troubleshooting  setting.  Troubleshooting  is,  in  part,  a 
complex  information-gathering  process  modified  by  a strategy  as  collected 
information  is  analyzed  by  the  troubleshooter.  The  information-gathering 
process  is  physically  represented  by  the  equipment  to  be  repaired,  the 
test  equipment  utilized  during  the  Information  gathering,  and  the  equip- 
ment maintenance  manuals. 

The  SPT  design  provided  the  troubleshooter  with  the  physical  cues 
(in  symbolic  form)  found  on  the  job,  allowing  the  troubleshooter  to  uti- 
lize the  strategy  of  his  choice. 

The  SPT  materials  included  the  following: 

1.  The  Equipment  Maintenance  Manual 

2.  Equipment  Photographs  to  Locate  Points  of  Test 

3.  Symbolic  Depiction  of  the  System  States  Available 
for  Troubleshooting 

4.  IIP  4 IOC  Voltmeter  Displays 

5.  HP  4 IOC  Function  and  Range  Displays 

6.  TEK  453A  Displays 

7.  TEK  453A  Set-Up  Choice  Displays 

8.  Conversion  Charts 

a.  Point  of  Test  by  System  State 

b.  System  State  by  HP  410C  (Range  and  Function) 

c.  System  State  by  TEK  453A  (Set-Up  Choice) 

9.  Front  Panel  Displays 

Subjects  taking  the  SPT  began  the  test  when  they  received  a write- 
up which  tersely  described  the  malfunction,  as  an  equipment  operator 
might.  The  subject  was  free  to  manipulate  front  panel  controls  and  note 
the  reaction  on  front  panel  displays  by  choosing  a system  state  (a  com- 
bination of  front  panel  control  settings)  from  an  illustration  and  then 
being  referred  to  an  illustration  which  pictured  the  resulting  front 
panel  displays.  Different  displays  were  available  for  each  choice  of 
system  state  and  for  each  malfunction. 

The  subject  was  also  free  to  make  electrical  tests  using  a VOM  or 
an  oscilloscope.  In  this  case,  the  subject  would  work  through  the  sym- 
bolic materials  by  choosing  from  illustrations  and  tables:  a point  of 
test,  a system  state,  a test  instrument,  and  a range  and  function  for 
the  test  instrument.  After  making  all  the  required  choices,  the  subject 
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would  then  be  referred  to  an  illustration  of  the  test  instrument  display 
in  which  he  would  obtain  the  test  result. 


Figure  7 illustrates  all  of  the  possible  use  paths  through  the  SPT 
materials.  Refer  to  the  figure  during  the  following  discussions  of  the 
test  materials  and  their  use. 


Locating  Points  of  Test.  Normally,  the  first  two  decisions  made  in 
a troubleshooting  sequence  are  (1)  where  the  test  should  be  made,  and 
(2)  how  the  equipment  controls  should  be  set.  The  SPT  sequence  was  be- 
gun by  identifying  the  desired  point  of  test.  This  was  done  in  two  steps. 
The  first  step  was  to  look  at  the  overall  equipment  photographs  (general 
locator)  and  identify  the  circuit  board  or  chassis  area  containing  the 
desired  components  or  point  of  test.  The  general  locator  referred  the 
subject  to  a second  (more  detailed)  photograph  which  depicted  the  compo- 
nents as  they  appear  in  the  equipment.  The  subject  located  the  desired 
point  of  test  on  this  photograph.  Each  point  of  test  was  designated  with 
an  arrow  and  identified  with  a code  number  (Figure  8).  Once  the  desired 
point  of  test  code  number  was  obtained,  the  subject  proceeded  either  to 
the  Point  of  Test/System  State  Conversion  Table  (for  front  panel,  AC, 

DC,  or  Scope  tests)  or  to  the  Voltmeter  Conversion  Table  for  Ohms  (for 
resistance  checks). 


Point  of  Test/System  State  Conversion.  This  conversion  was  accom- 
plished in  a simple  matrix.  Refer  to  Figure  9 for  the  matrix  format. 


The  matrix  was  entered  in  the  left  column  with  the  previously  deter- 
mined Point  of  Test  I.D.  number.  The  I . D.  numbers  were  in  numerical  or- 
der to  make  finding  any  given  number  easy.  Once  the  I.D.  number  row  was 
located,  the  subject  specified  a system  state  by  choosing  one  of  the  il- 
lustrated system  states  and  finding  its  code  number  in  the  remaining  col- 
umn headings. 


The  subject  read  down  the  desired  column  to  the  intersection  of  the 
test  point  I.D.  number  row.  That  cell  contained  the  system-state  guide 
number  required  to  access  a front  panel  display,  or  an  AC,  DC,  or  Scope 
measurement.  For  example,  if  the  point  of  test  was  a front  panel  dis- 
play (Ml  on  the  HP  652A) , the  subject  would  go  to  the  front  panel  displays 
section.  If  not,  he  would  then  specify  the  nature  of  the  desired  test 
(AC,  DC,  waveform)  by  using  the  conversion  table  associated  with  the  test 
instrument  and  function  (Voltmeter  Conversion  Tables:  AC,  DC+,  DC-;  or 
Scope  Conversion  Table) . 


Voltmeter  AC  and  DC  Conversions.  Refer  to  Figure  10  during  the  fol- 
lowing discussion.  The  subject  used  this  form  just  as  he  did  the  Point 
of  Test/System  State  matrix.  (Note:  Only  one  point  of  test  was  speci- 
fied for  the  AC  and  DC  conversions.  The  other  point  of  test  was  under- 
stood to  be  common  or  circuit  return.)  The  subject  selected  the  appro- 
priate range  setting  by  viewing  an  illustration  and  obtaining  a code 
number  for  function  and  range.  These  code  numbers  represented  the  col- 
umn heads.  The  number  at  the  intersection  of  the  system  state/test 
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Figure  9.  Point  of  Test/System  State  Conversion  Table 
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point  and  range  and  function  was  the  guide  number  to  a meter  display 
illustrating  the  test  result. 


Figure  10.  Voltmeter  Conversion  Table:  AC 

Voltmeter  Ohms  Conversion.  The  ohms  conversion  was  similar  to  the 
AC  and  DC  versions,  except  that  the  first  column  contained  dual  entries 
for  the  point  of  test  (Figure  11). 

The  dual  point  of  test  columns  were  identified  by  ohmmeter  leads. 
This  permitted  forward  and  reverse  resistance  readings  on  diode  and  tran- 
sistor junctions  since  all  points  of  test  were  covered  sequentially 
(e.g.,  T1-T49  was  also  included  as  T49-T1). 

All  entries  in  the  dual  column  were  in  first-column  numerical  order 
for  easy  access. 

Front  Panel  Displays.  The  front  panel  displays  were  illustrations 
of  the  subject  equipment  with  switches  set  appropriately  for  selected 
system  states,  and  with  the  appropriate  meter  face  or  display  given  the 
system  state  and  the  malfunction.  The  desired  display  was  identified  on 
the  Point  of  Test/System  State  Conversion  Table. 
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VOLTMETER  CONVERSION  TABLE:  OHMS 


POINTS 

OF  TEST 

COM 

S 

25 

26 

27 

28 

29 

T1 

T49 

V23 

V7 

V6 

V4 

V18 

T2 

T58 

Vl4 

Vl9 

V3 

V5 

V23 

T3 

T47 

V19 

V8 

V7 

V3 

V20 

T4 

T63 

V18 

V8 

V6 

V20 

V20 

T5 

T110 

V17 

Vll 

V6 

V3 

V20 

T6 

T12 

V9 

VlO 

V14 

V2 

V20 

T6 

T14 

V3 

V2 

Vll 

V9 

V7 

T6 

T19 

V17 

V15 

V25 

V20 

V20 

T7 

T19 

V4 

V3 

V2 

V20 

V20 

T7 

T20 

V5 

V9 

V3 

V8 

V12 

T7 

T24 

V5 

V3 

V2 

Vll 

V20 

^ T9 

"^\V20 

S' 

Figure  11.  Voltmeter  Conversion  Table:  Ohms 

Voltmeter  Displays.  There  was  a series  of  meter  readings  associated 
with  Voltmeter  Conversion  Guide  Numbers  (Figure  12).  The  correct  display 
was  the  one  carrying  the  guide  number  obtained  for  a specific  check  in 
the  Voltmeter  Conversion  Table. 

Oscilloscope  Displays.  The  scope  waveforms  were  identified  by  a 
guide  number  obtained  in  the  Scope  Conversion  Table.  The  scope  conver- 
sion guide  numbers  were  associated  with  individual  waveforms,  just  as 
the  meter  displays  were.  See  Figure  13  for  a sample  of  the  scope  dis- 
plays. 


SPT  Materials  Development 

The  information  contained  in  the  sequential  matrices  was  obtained  by 
actually  performing  the  indicated  measurements.  Three  items  each  were 
developed  for  the  HP  652A  and  the  TEK  453A  for  a total  of  six.  The  sys- 
tem states  and  the  scope  and  VOM  ranges  and  functions  to  be  included  were 
specified  for  each  piece  of  equipment  prior  to  collecting  the  measure- 
ments and  remained  the  same  for  all  items  on  a particular  piece  of  prime 
equipment. 


The  previously  designed  and  tested  malfunctions  (items)  were  in- 
stalled individually.  Measurements  were  made  with  both  the  scope  and 
VOM  at  each  point  of  test,  in  each  system  state,  and  at  each  specified 
test  equipment  range  and  function. 

The  number  of  tests  to  be  made  was  also  very  large.  Several  mea- 
sures were  taken  to  reduce  the  required  number  of  measurements.  These 
measures  included  taking  all  readings  with  all  test  equipment  with  no 
malfunction  installed,  then  installing  a malfunction  and  retaking  only 
the  affected  readings.  The  number  of  readings  was  also  reduced  by  elim- 
inating those  readings  which  were  irrelevant,  such  as  VOM  RMS  values  in 
DC  circuits;  in  most  cases,  RMS  values  were  provided  only  in  the  power 
supply  circuits.  Waveform  coverage  was  limited  to  those  circuits  pro- 
ducing periodic  signals.  With  the  VOM,  it  was  not  necessary  to  repeat 
measurements  on  different  voltage  ranges  because  of  a sufficiently  high 
meter  input  impedance.  Il  was  necessary,  however,  to  measure  resistances 
in  semiconductor  circuits  on  each  range  because  of  different  ohmmeter 
output  impedance  on  each  range.  By  limiting  measurements  to  relevant 
parameters,  the  number  of  required  measurements  was  reduced  from  a few 
hundred  thousand  to  a few  thousand. 


Preliminary  Tryout 

The  preliminary  tryout  was  held  to  debug  the  test  materials  and  in- 
structions. The  JST  and  SPT  test  materials  used  were  for  the  HP  652A. 

The  tryout  was  conducted  using  an  industrial  electronics  technician 
with  qualifications  roughly  equivalent  to  those  of  an  experienced  PMEL 
technician  or  instructor.  The  subject  completed  six  JST  items  (the  three 
designated  for  the  final  testing  and  three  alternates)  and  three  SPT  items. 

Results.  The  subject  correctly  solved  all  JST  and  SPT  problems.  The 
subject's  performance  time  on  the  JST  was  within  the  originally  planned 
60  minutes,  while  SPT  performance  time  was  significantly  longer  than  60 
minutes  on  two  of  the  three  items. 

The  original  answering  scheme  was  judged  too  difficult,  and  the  re- 
sults were  too  unreliable.  The  answering  scheme  required  the  subject  to 
identify  the  malfunctioning  stage  by  listing  the  components  making  up  the 
stage.  This  process  was  too  judgmental  to  provide  reliable  results  with- 
out a large  amount  of  pretest  training  in  stage  identification. 

The  subject  worked  on  the  first  symbolic  item  approximately  80  min- 
utes before  the  administrator  discovered  an  instruction  omission  and  an 
SPT  materials  defect.  Once  the  complete  instructions  were  provided  and 
the  defect  corrected,  the  subject  completed  the  item  satisfactorily. 

The  symbolic  materials  did  not  address  the  scope  probe  attenuation  fac- 
tor. This  accounted  for  the  subject's  difficulties  with  the  first  sym- 
bolic item.  The  original  materials  assumed  that  the  normal  (10X)  probe 
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would  be  used;  however,  the  probe  was  not  pictured  in  the  scope  range 
and  function  choice  illustrations. 


Subject's  Comments.  The  subject  observed  that  one  reason  for  longer 
performance  times  on  the  SPT  was  the  limited  number  of  test  points  pro- 
vided. Test  design  provided  for  one  physical  location  for  each  electri- 
cal point  of  test,  whereas  in  the  real  equipment  there  may  be  several 
alternate  physical  locations  for  the  same  point  of  test. 

The  tryout  technician  preferred  to  use  the  oscilloscope  for  all  diag 
nostic  measurements,  but  was  unable  to  do  so  since  the  SPT  materials  were 
designed  to  minimize  test  equipment  overlap  whenever  possible. 


The  subject  noticed  some  solder  flux  on  the  circuit  boards  in  the 
area  of  the  installed  malfunction.  The  subject  was  not  able  to  identify 
thi  malfunction  visually,  but  to  some  extent  received  confirmation  of 
being  in  the  right  "ballpark." 


SPT  Design  Modifications 

As  a result  of  the  tryout,  some  features  of  the  SPT  instructions  and 
materials  were  modified.  The  changes  are  discussed  below. 

Answering  Scheme.  The  problems  associated  with  stage  identification 
were  eliminated  by  providing  a standard  set  of  stage  definitions.  The 
standard  stage  definitions  were  the  maintenance  manual  schematics  repro- 
duced with  partitioning  lines  drawn  around  individual  stages.  Each  stage 
was  assigned  a reference  number  for  answering  purposes.  A subsequent 
tryout  demonstrated  that  the  answering  problem  was  solved  and  that  time 
to  solution  was  also  reduced. 

Oscilloscope  Range  and  Function  Drawings.  These  drawings  were  mod- 
ified to  include  the  probe.  The  probe  was  positioned  so  that  the  atten- 
uation factor  was  visible  in  the  illustration. 

Points  of  Test.  This  problem  was  considered,  but  no  changes  were 
made  to  the  test  coverage.  There  was  no  doubt  that  by  increasing  the 
number  of  points  of  test  the  materials  would  be  easier  to  use;  however, 
finding  desired  points  of  test  in  actual  equipment  is  often  just  as 
difficult. 

Test  Equipment  Use.  The  intent  was  to  provide  point  of  test  value 
information  with  the  easiest- to-operate  test  equipment  capable  of  accu- 
rately reading  the  parameter  of  interest.  The  rationale  was  that  using 
more  dif f icult-to-operate  test  equipment  than  is  required  to  obtain  a 
measurement  is  a more  error-likely  process. 

In  a few  instances,  the  technical  manual  contained  a measurement  at 
a point  of  test  not  provided  in  the  materials.  In  these  cases,  the  mate- 
rials were  expanded  to  contain  this  overlap.  No  other  changes  were  made. 
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Malfunction  Insertion  Techniques.  New  circuit  boards  were  installed 
to  replace  the  old  ones  which  had  evidence  of  soldering.  Malfunctions 
in  the  oscilloscope  JST  were  selected  for  plug-in  components.  Spare 
components  were  obtained  and  failed.  During  testing,  bad  components 
could  be  substituted  for  good  ones  by  simply  unplugging  the  good  one 
and  plugging  in  the  bad. 

Where  there  were  no  plug-in  components  in  the  oscillator,  additional 
wires  were  added  to  the  chassis  wiring  harness  and  connected  to  the  cir- 
cuit boards.  These  wires  and  connections  were  not  visible.  This  wiring 
was  then  connected  to  switches  which,  when  operated,  installed  or  removed 
the  malfunctions.  These  switches  were  placed  inside  a closed  container 
where  their  positions  and  effects  could  not  be  seen  or  detected  by  the 
subjects . 


Test  Administration 


Study  Context 

Troubleshooting  performance  was  tested  on  two  pieces  of  PMEL  equip- 
ment: the  HP  652A  test  oscillator  and  the  TEK  453A  oscilloscope.  Each 

test  contained  three  items,  with  an  item  being  the  isolation  of  a single 
malfunction  to  the  stage  level  of  penetration.  Acceptable  performance 
was  criterion-referenced  (go/no-go)  and  time  to  solution  was  measured. 

Criterion  and  symbolic  tests  were  given  to  31  subjects:  15  appren- 
tice level  technicians  (PMEL  students),  and  16  journeyman  level  techni- 
cians (PMEL  course  instructors). 


Test  Administrator  Role 

The  administrator's  role  encompassed  a variety  of  activities  in- 
cluding: 

1.  Providing  a project  briefing. 

2.  Providing  test  instructions. 

3.  Checking  out  the  Lme  equipment  before  installing 
JST  problems. 

4.  Assigning  and  installing  JST  problems. 

5.  Monitoring  JST  and  SPT  activity. 

6.  Scoring  JSTs  and  SPTs. 
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7.  Providing  subjects  with  SPT  briefing. 

8.  Demonstrating  SPT  practice  problems. 

9.  Monitoring  additional  SPT  practice. 

10.  Assigning  SPT  problems. 

JST  Equipment  Checkout  and  Problem  Installation.  The  administrator 
verified  proper  operation  of  the  prime  equipment  before  installing  a JST 
problem.  The  administrator  also  verified  the  malfunction  after  installing 
a problem. 

Student-induced  malfunctions  (e.g.,  shorting  test  points  to  ground) 
were  handled  in  one  of  two  ways: 

1.  If  the  administrator  noted  the  exact  moment  the 
student  caused  a malfunction,  the  problem  timer 
was  stopped  and  the  subject  given  a break  while 
the  administrator  repaired  the  equipment.  Once 
the  equipment  was  repaired,  the  student  resumed 
and  the  timer  was  restarted. 

2.  If  the  administrator  detected  a student-caused 
problem  after  its  introduction,  the  student  was 
given  a break  and  the  equipment  was  repaired. 

The  timer  was  then  reset  and  the  student  started 
the  problem  over. 

SPT  Practice  Problems.  SPT  sample  materials  were  developed  to  be 
used  by  the  administrator  to  demonstrate  the  SPT  concept.  The  sample 
materials  contained  two  short  problems  which  the  instructor  solved  in 
a demonstration  for  the  SPT  subjects.  The  sample  problem  materials  are 
contained  in  Appendix  A of  this  report.  The  SPT  subjects  were  then  given 
an  actual  SPT  problem  on  the  HP  652A  as  a practice  item.  The  subjects 
solved  this  problem  with  no  time  limit,  since  the  practice  item  was  in- 
tended to  familiarize  the  subjects  with  the  materials.  The  sample  prob- 
lems and  practice  problem  were  prerequisites  for  participation  in  the 
SPT. 

Problem  Order  Assignments.  Since  the  JST  and  SPT  items  for  an  indi- 
vidual piece  of  prime  equipment  were  the  same,  the  order  of  presentation 
for  each  subject  between  JST  and  SPT  was  varied.  The  order  of  presenta- 
tion between  subjects  was  also  varied  to  prevent  association  of  answer 
with  presentation  order,  and  to  insure  that  subjects  were  not  working 
on  the  same  item  at  the  same  time  during  data  collection. 

Figure  14  is  a sample  problem  assignment  record.  It  contains  the 
item  presentation  order  combinations  used. 
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TESTEE  NAME 


JOB  SAMPLE  SYMBOLIC 


*1, 

2, 

3 

2, 

3, 

1 

2, 

3, 

1 

*3, 

2, 

1 

*3, 

1, 

2 

1, 

2, 

3 

2, 

1, 

3 

*2, 

3, 

1 

*1, 

3, 

2 

1, 

2, 

3 

3, 

2, 

1 

*3, 

1, 

2 

*2, 

3, 

1 

2, 

1, 

3 

1, 

2, 

3 

*1, 

3, 

2 

*3, 

1, 

2 

1, 

2, 

3 

2, 

1, 

3 

*3, 

2, 

1 

*1, 

3, 

2 

3, 

2, 

1 

3, 

2, 

1 

*2, 

3, 

1 

*2, 

3, 

1 

2, 

1, 

3 

1, 

2, 

3 

*3, 

1, 

2 

*3, 

1, 

2 

1, 

3, 

2 

♦Subject  begins  with  this  test  form. 


Figure  14.  Problem  Assignment  Sheet 
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Test  Site  Layout 

Refer  to  Figure  15  for  the  test  site  floor  plan.  The  JST  carrels 
were  created  by  separating  two  work  benches  with  a portable  visual  bar- 
rier. The  SPT  carrels  each  consisted  of  two  30  by  60  inch  tables.  The 
SPT  carrels  were  also  separated  with  a portable  visual  barrier. 


In  addition  to  the  prime  equipment,  each  JST  carrel  was  supplied 
with  an  HP  4 10C  transistorized  voltmeter  and  a Tektronix  453  scope  to  be 
used  as  troubleshooting  test  equipment.  The  prime  equipment  had  all  ac- 
cess covers  removed,  eliminating  the  time  required  to  remove  them  and 
eliminating  the  subjects'  need  for  hand  tools.  Each  carrel  also  con- 
tained a clock  with  a start/stop  switch  to  serve  as  a timer. 


SCOPE  JST 
CARREL 


OSCILLATOR  JST 
CARREL 


SCOPE  SPT 
CARREL 


ADMINISTRATOR'S 

DESK 


OSCILLATOR 
SPT  CARREL 


Figure  15.  Test  Site  Floor  Plan 
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Data  Collected 


The  data  were  collected  on  two  sources:  an  answer  card  (Figure  16) 
and  a protocol  record  (Figure  17). 


The  test  administrator  started,  stopped,  and  reset  the  timers  in 
each  carrel.  The  timer  was  started  when  a subject  began  a problem  and 
was  stopped  when  the  problem  was  completed.  The  test  administrator  noted 
the  elapsed  time  on  the  answer  card.  The  subject  entered  his  choice  of 
defective  stage  on  the  answer  card. 


The  subjects  completed  the  JST  or  SPT  Record  as  they  worked  through 
the  problem  solution.  The  subject  recorded  each  diagnostic  check  and 
the  point  tested  within  the  equipment  on  this  form. 


ANSWER  CARD 


John  Doe 


9-22-77 


Social  Security  No. 


Problem  No.. 


Complaint 


. ^ ...  Total  Time  37  min. 

MeTcA  indicate  0 in  normal. 


Defective  Stage  No. 


Figure  16.  Answer  Card 


Scorinf 


The  test  administrator  checked  the  defective  stage  choice  and  indi- 
cated either  correct  or  incorrect.  The  elapsed  time  was  considered  in 
the  analyses  as  time  to  completion. 


The  test  records  yielded  steps  to  completion  by  counting  the  number 
of  diagnostic  checks  made. 


The  accuracy  (correct  versus  incorrect),  time  to  completion,  and 
steps  to  completion  raw  data  are  included  in  Appendices  B,  C,  and  D, 
respectively. 
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Problem  No. 


RESULTS 


Introduction 


The  following  section  examines  the  JST  and  SPT  results.  The  results 
are  first  viewed  combined;  i.e.,  all  subjects,  all  problems.  Secondly, 
the  subdivisions  of  subjects  and  treatments  are  examined  individually 
to  determine  their  effects  on  the  outcome. 


Summary  Output  Results 


The  three  measures  of  comparison  between  JST  and  SPT  performance  are 
accuracy  (correct  or  incorrect),  time  to  completion,  and  steps  to  comple- 
tion. Accuracy  data  were  expressed  in  mean  percent  correct,  and  time 
data  were  expressed  in  minutes  to  completion. 


Accuracy 


JST  versus  SPT 


There  are  two  important  aspects  of  the  accuracy  results.  The  first 
is  that  the  overall  accuracy  (correct  versus  incorrect  solution)  was  very 
low;  subjects  failed  approximately  two  problems  out  of  every  three,  par- 
tially because  only  one  try  was  permitted.  The  second  is  that  there  was 
a total  of  six  items  for  each  treatment  group  (JST/SPT)  resulting  in  only 
six  data  points  to  produce  the  accuracy  means.  The  resulting  step  func- 
tion between  scores  possible  (i.e.,  16,  34,  30,  67,  83,  100)  tended  to 
suppress  accurate  correlations. 

Figure  18  illustrates  the  accuracy  grand  means  for  JST  and  SPT.  Cor- 
relation between  mean  scores  was  +.384  which  is  significant  at  the  .023 
level . 

In  his  discussion  of  measuring  association  in  ordered  classes,  Hays 
(1963)  said  ".  . .the  value  of  the  t statistic  itself  does  not  seem  to 
have  a very  simple  interpretation  when  ties  are  present  in  either  rank- 
ing. This  difficulty  is  removed  if  one  uses  the  y statistic  suggested 
by  Goodman  and  Kruskal  (1954)  specifically  for  data  arranged  in  ordered 
classes  [p.  655]."  The  formula  for  the  Goodman-Kruskal  (GK)  gamma  is: 

Nc-Nd 

Nc+Nd 

where  Nc  is  the  number  of  concordant  sets  and  Nd  is  the  number  of  discor- 
dant sets.  This  statistic  is  interpreted  like  a correlation  coefficient. 


GK  gammas  are  reported  for  comparison  with  the  accuracy  correlations  since 
the  gammas  are  more  sensitive  to  associated  relationships. 


Figure  18.  Accuracy  Grand  Means 

In  this  case,  looking  at  pass-fail  results  for  each  set  of  JST/SPT 
problems,  the  GK  gamma  is  +.376  which  closely  parallels  the  correlation 
coefficient  of  +.384. 


Time  to  Completion:  JST  versus  SFT 

Figure  19  illustrates  the  time  to  completion  grand  means  of  JST  and 
SPT  tests.  Correlation  between  times  was  +.588  which  is  significant 
to  the  .0005  level.  Time  to  completion  was  limited  at  the  high  end  by 
a maximum  limit  of  60  minutes.  Subjects  that  did  not  solve  the  problem 
in  60  minutes  ("timed  out")  were  also  given  a fail  on  the  accuracy  mea- 
sure. There  was  a total  of  28  "time  outs"  out  of  372  trials.  Approx- 
imately 7.5  percent  of  the  data  was  influenced  by  "time  outs."  The 
"time  out"  effects  on  the  grand  means  are  minimal,  since  they  are  less 
than  30  minutes.  The  restriction  In  range  probably  suppressed  slightly 
the  obtained  correlations. 
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Figure  20  illustrates  the  steps  to  completion  grand  means  for  JST 
and  SPT  tests.  Correlation  between  scores  was  +.356  which  is  signifi- 
cant at  the  .025  level.  Steps  to  completion  is  viewed  as  the  least 
significant  output  measure  for  demonstrating  similarity  since  it  is 
also  related  to  time  to  completion  (i.e.,  the  greater  the  number  of 
steps  to  completion,  the  longei  it  will  take  to  perform  them).  It  is 
interesting  to  note  the  mean  time  per  step,  using  the  data  from  Figures 
19  and  20  is  4.54  minutes  per  JST  step  and  6.47  minutes  per  SPT  step. 

The  SPT  steps  took  on  the  average  42-1/2  percent  longer  to  perform  than 
the  JST  steps.  This  fact  was  sensed  and  mentioned  by  many  of  the  sub- 
jects in  their  comments  on  the  SPT.  Working  through  the  SPT  photographs 
and  sets  of  matrices  appears  to  take  longer  than  performing  a check  on 
the  actual  i uipment. 


Figure  20.  Steps  to  Completion  Grand  Means 


Factors  Affecting  the  Summary  Accuracy  Correlation 


As  one  might  expect,  several  subject  and  treatment  features  had 
some  effect  on  the  summary  accuracy  correlation.  The  significant  effects 
are  discussed  below. 


Accuracy 


Students  versus  Instructors 


Figure  21  illustrates  the  accuracy  grand  means  for  students  and  in- 
structors for  both  JSTs  and  SPTs.  The  JST/SPT  accuracy  correlation  for 
instructors  was  +.628  which  is  significant  at  the  .0005  level.  The 
JST/SPT  accuracy  correlation  for  students  was  -.041  which  is  rot  signi- 
ficant. The  low  student  correlation  accounts  for  the  abnormally  low 
value  of  the  overall  accuracy  correlation.  However,  the  instructor 
GK  gamma  was  +.31  while  the  student  GK  gamma  was  +.44.  The  GK  gamma 
depicts  a more  consistent  level  of  performance  between  these  two  groups 
than  the  correlations  do. 


Student  JST  results  matched  SPT  results  (right  on  JST  and  SPT  or 
wrong  on  JST  and  SPT)  72  percent  of  the  time,  while  instructor  results 
matched  65.6  percent  of  the  time. 
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Figure  21.  Accuracy  Grand  Means  for  Students  and  Instructors 

Accuracy:  Oscillator  Problems  versus  Scope  Problems 

Figure  22  illustrates  the  accuracy  grand  means  for  scope  and  oscil- 
lator for  both  JSTs  and  SPTs.  The  correlation  for  oscillator  JST  and 
SPT  items  was  +.29  which  is  not  significant.  The  corresponding  scope 
correlation  was  +.48  which  is  significant  at  the  .005  level.  The  GK 
gammas  are  as  follows:  oscillator  +.31  and  scope  +.44.  The  oscillator 
JST  results  matched  SPT  results  61  times  out  of  93  possible  or  65.5 
percent  of  the  time.  The  scope  JST  results  matched  SPT  results  69  times 
out  of  93  possible  or  74  percent  of  the  time. 

Oscillator  Accuracy:  Students  versus  Instructors 

Figure  23  illustrates  the  oscillator  accuracy  grand  means  for  stu- 
dents and  instructors. 

The  correlation  for  student  JST/SPT  performance  was  +.19  which  is 
not  significant.  The  correlation  for  instructors  was  +.34  which  also 
is  not  significant. 

The  GK  gammas  are  as  follows:  students  +.47,  instructors  +.16.  The 
GK  results  contradict  the  correlations,  which  suggests  that  the  correla- 
tions may  not  be  as  appropriate  as  the  gammas  for  judging  the  similarity 
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of  performance  in  this  case.  Student  results  matched  33  times  out  of  45 
trials  or  73  percent  of  the  time.  Instructor  results  matched  28  times 
out  of  48  trials  or  58  percent  of  the  time. 


Scope  Accuracy 


Students  versus  Instructors 


Figure  24  iilustrates  the  scope  accuracy  grand  means  for  students 
and  Instructors. 

The  correlation  for  student  JST/SPT  performance  was  +.42  which  is 
significant  at  the  .05  level  of  confidence.  The  correlation  for  instruc- 
tors was  +.54  which  is  significant  at  the  .025  level  of  confidence. 

The  GK  gammas  are  as  follows:  students  +.42,  instructors  +.46. 
Student  results  matched  in  32  of  45  trials  or  71  percent  of  the  time. 
Instructor  results  matched  35  out  of  48  trials  or  73  percent  of  the 
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Figure  24.  Scope  Accuracy  Means  for  Students  and  Instructors 
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Factors  Affecting  the  Summary  Time  Correlation 


It  may  be  difficult  to  conclude  that  performance  time  is  similar  for 
the  different  treatments  (JST  and  SPT)  based  on  time  alone  when  the  task 
to  be  performed  (at  the  step  level)  is  not  controlled.  That  is,  the  tech- 
nician may  use  different  numbers  of  steps  and  in  different  order  to  solve 
each  presentation  of  the  problem.  In  this  study,  the  task  was  not  con- 
trolled; however,  the  protocol  data  suggest  that  the  same  strategies  were 

followed  in  both  test  formats.  Correlations  for  student  and  instructor  | 

time  to  completion  for  JST  and  SPT  are  as  follows:  students  +.72,  sig- 
nificant to  the  .005  level  of  confidence;  instructors  +.63,  significant 
to  the  .0005  level  of  confidence.  Correlations  between  oscillator  and 
scope  JST  and  SPT  problems  are  as  follows:  oscillator  +.56,  significant 
to  the  .0005  level  of  confidence;  scope  +.39,  significant  to  the  .025 
level  of  confidence. 

The  summary  time  correlations  were  quite  high  and  significant.  If 
one  accepts  that  the  tasks  performed  were  similar  on  the  basis  of  the 
protocol  analysis  of  strategy,  then  one  must  conclude  that,  time-wise, 
performance  was  similar  regardless  of  presentation  mode. 


Presentation  Order  Effects 

Each  subject  received  each  test  item  twice,  once  as  a JST  problem 
and  once  as  an  SPT  problem.  It  was  possible  that  performance  on  the 
second  presentation  could  be  affected  by  the  previous  exposure.  First 
exposure  as  used  here  is  equivalent  to  the  test  form  (JST  or  SPT)  ini- 
tially worked  by  a subject.  The  second  exposure  is  equivalent  to  a 
retest  in  the  other  test  form. 

In  a test/retest  situation  such  as  this  using  the  same  limited 
numbers  of  items,  it  was  reasonable  to  expect  some  improvement  on  the 
retest.  The  expectation  is  that  the  initial  encounter  will  produce 
some  learning;  however,  on  such  a small  test  there  is  the  fear  of  test 
compromise.  The  following  discussion  reviews  the  effects  of  presenta- 
tion order  on  each  of  the  performance  measures. 


Presentation  Order  and  Accuracy 

Figure  25  compares  the  accuracy  means  of  first  and  second  presenta- 
tion scores  in  several  ways.  First  and  second  exposures  for  all  subjects 
and  both  test  forms  are  compared  with  the  results  of  the  student  and 
instructor  groups.  The  student  and  instructor  results  are  compared  with 
the  SPT-first  exposure  contrasted  against  the  JST-first  exposure, 
v 
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The  data  in  each  category  of  first-  and  second-exposure  comparison 
were  evaluated  with  the  chi-square  statistic.  Chi-square  was  chosen 
in  preference  to  a t-test  since  the  accuracy  data  were  seriously  skewed 
toward  the  low  end  of  the  distribution  (see  Guilford,  1956,  p.  221). 

This  low  end  distribution  is  accounted  for  by  the  overall  mean  score 
of  approximately  38  percent. 

Using  the  chi-square  evaluation,  all  of  the  first-  versus  second- 
presentation  differences  depicted  in  Figure  25  are  significant  beyond 
the  .0001  level. 

The  pooled  data  comparison  indicated  a seven  percent  improvement  be- 
tween first  and  second  exposures,  which  is  far  less  than  one  would  expect 
if  the  test  had  been  compromised.  The  marked  differences  between  first- 
and  second-exposure  results  for  students  suggest  that  the  students  learned 
about  the  equipment,  its  technical  data,  and/or  troubleshooting  as  a re- 
sult of  first-test  exposure.  The  student  improvement  in  accuracy  on  the 
second  presentation  regardless  of  format  was  approximately  13  percent. 

One  would  expect  the  instructors  to  be  far  more  familiar  with  the  equip- 
ment and  to  have  adopted  some  troubleshooting  philosophy  and  that  one  ad- 
ditional exposure  to  the  equipment  would  produce  little  change  in  their 
performance.  This  hypothesis  is  partly  borne  out  by  the  instructor  re- 
sults. Indeed,  while  the  instructor  differences  were  significant,  the 
increment  of  change  was  many  times  less  than  that  of  the  students. 

While  the  student  SPT- first  means  were  somewhat  lower  than  their 
JST-first  means,  it  is  important  to  note  the  13  percent  increment  of 
improvement  using  the  SPT  first. 


Figure  25.  Presentation  Order  Effects  on  Accuracy 


(V 


55 


**••*■**•  1 ■ - - " i.  »«<■,  'oit.Wtna 


An  interesting  comparison  can  be  made  between  the  student  cate- 
gories of  SPT-first  versus  JST-flrst.  The  SPT-first  group's  time  to 
completion  on  the  SPT  was  faster  than  the  JST-flrst  group's  time  to 
completion  on  either  test. 

The  students  in  the  SPT-first  group  also  had  a faster  time  to  com- 
pletion when  they  took  the  JST  than  cither  of  the  instructor  groups.  A 
statistical  comparison  of  the  SPT-first  student  .1ST  results,  with  the 
shortest  instructor  JST  times  showed  no  significant  differences.  A 
t-test  revealed  a t of  0.35  which  is  not  significant  at  the  .05  level. 

A final  point  of  interest  is  the  instructor  JST  time  scores;  re- 
gardless of  presentation  order,  their  time  to  completion  means  were 
within  0.1  minutes.  A t-test  confirmed  that  there  were  no  significant 
differences  in  these  means.  The  t-statistic  was:  t = .024  which  is  not 
significant  at  the  .05  level.  This  finding  helps  support  the  assertion 
made  in  the  section  on  presentation  and  accuracy  that  one  more  exposure 
to  the  equipment,  for  instructors  should  produce  little  change  in  the 
instructor  performance. 


Presentation  Order  and  Steps  to  Completion 

Figure  27  compares  the  steps  to  completion  means  for  the  first  and 
second  exposures  in  the  same  categories  presented  for  accuracy  and  time 
in  Figures  25  and  26. 

The  data  were  evaluated  using  t-tests.  The  t-tests  indicated  no 
significant  differences  between  the  means  in  all  categories  except  in- 
structor SPT-first. 

The  t-test  results  are: 

Pooled:  t = .34,  not  significant  at  the  .05  level 
Student  SPT-first:  t = 1.26,  not  significant  at  the  .05  level 

Student  JST-first:  t = 1.46,  not  significant  at  the  .05  level 

Instructor  SPT-flict:  t = -1.78,  not  significant  at  the  .05  level 

Instructor  JST-first:  t = 3.47,  significant  at  the  .001  level 

Again,  the  hypothesis  for  test  compromise  stated  for  steps  to  com- 
pletion would  be  a significantly  Fewer  number  of  steps  to  completion  in 
the  second  encounter.  While  the  difference  was  not  significant,  the 
pooled  data  indicated  a slightly  lower  number  of  steps  to  completion 
for  the  second  presentation. 


lit  2nd 


1st  2nd  1st  2nd 
STUDENTS 


1st  2nd  1st  2nd 
INSTRUCTORS 


Figure  27.  Presentation  Order  Effects  on  Steps  to  Completion 

The  protocol  data  does  suggest  that  a few  competent  troubleshooters 
followed  a systematic  strategy  to  solve  a problem  on  the  first  exposure, 
but  went  right  to  the  suspect  component  and  confirmed  that  it  was  the 
problem  in  the  second  exposure.  However,  the  combined  steps  to  comple- 
tion data  suggest  that  this  was  generally  not  the  case. 

Analysis  of  the  student  and  instructor  breakdowns  in  Figure  27  re- 
veals one  other  important  fact:  In  every  case  there  were  fewer  steps 
to  completion  in  the  SPT  format  than  in  the  JST  format.  This  may  be  a 
function  of  SPT  structure  or  volume  of  materials  forcing  more  deliberate 
behavior. 


Protocol  Data 


The  protocol  data  consisted  of  the  list  of  steps  (and  points  of  test) 
performed  by  the  subject  while  attempting  to  solve  each  problem.  The 
protocol  records  were  analyzed  for  troubleshooting  process  information 
and  were  compared  with  an  optimal  solution.  The  comparison  was  made  on 
a data  flow  drawing  of  the  equipment  at  the  stage  level  (see  Figure  28 
for  an  example) . 


OSCILLATOR  BOARD 


Figure  28.  Protocol  Map 


The  legitimate  search  field  was  indicated  in  the  clear  area.  Sub- 
ject's checks  in  the  shaded  portions  were  classed  as  irrelevant  to  the: 
problem.  The  optimal  solution  was  charted  on  the  drawing  both  in  te  ms 
of  check  location,  type  of  check,  and  check  sequence.  Each  subject's 
checks  were  also  charted  on  the  data  flow  drawing  in  the  same  manner. 
This  permitted  a direct  comparison  with  the  optimal  solution. 

The  protocol  analyses  classified  troubleshooting  behavior  in  terms 
of  approach  to  solving  the  problems.  Each  solution  was  compared  to  the 
following  criteria: 

1.  Symptom  Pattern  Analysis — isolated  suspect  subset 
of  possible  causes  from  malfunction  description. 

2.  Input  Checks — checked  input  to  suspect  subset 
of  data  flow. 

3.  Output  Checks — checked  output  of  suspect  subset 
of  data  flow. 

4.  Internal  Checks — checked  within  suspect  subset 
of  data  flow. 

5.  Power  Supply  Checks — checked  unit  common  power 
supplies. 

6.  Irrelevant  Checks — checked  outside  the  subset  of 
suspect  components. 


Protocol  Results 


Correlating  the  Symptom  Pattern  Analysis  category  results  for  all 
JST/SPT  tests  produced  a correlation  coefficient  of  +.962  which  is  sig- 
nificant at  the  .0005  level  of  confidence.  This  fact  very  strongly  sug- 
gests that  most  subjects  used  the  same  (and  an  appropriate)  strategy  to 
find  the  problem,  regardless  of  treatment. 

The  internal  checks  and  output  classes  produced  correlation  coeffi- 
cients of  +.926  and  +.916  respectively.  The  input  check  category  pro- 
duced a correlation  coefficient  of  +.90.  These  correlations  are  all 
significant  at  the  .005  level  of  confidence.  This  suggests  that  regard- 
less of  treatment,  most  subjects  made  an  attempt  to  identify  the  problem 
by  testing  within  the  area  of  the  equipment  containing  the  malfunction. 

Power  supply  checks  are  often  a routine  part  of  a troubleshooter's 
approach  since  they  tend  to  be  a common  link  to  the  remainder  of  the 
equipment.  The  power  supply  check  category  produced  a correlation  of 
+.748  between  JST  and  SPT  treatments,  which  is  significant  at  the  .025 
level. 
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In  the  irrelevant  check  category,  comparison  of  the  JST/SPT  treat- 
ments produced  a correlation  coefficient  of  +.469,  which  is  not  signifi- 
cant. 


Problem  Difficult} 


The  following  analysis  was  performed  to  evaluate  the  problem  dif- 
ficulty algorithm  which  was  developed  during  this  study  as  an  aid  in 
selecting  problems.  The  percentage  of  incorrect  solutions  was  calcu- 
lated for  each  test  item.  These  scores  were  correlated  with  the  level 
of  difficulty  numbers  calculated  using  the  problem  difficulty  algorithm. 
The  correlation  for  the  combination  of  all  JST  and  SPT  results  is  +.567, 
which  is  not  significant  with  only  six  data  pairs.  The  correlation  for 
only  JST  results  is  +.642  which  is  significant  at  the  .05  level  of  con- 
fidence. The  correlation  for  only  SPT  results  is  +.467  which  is  not 
significant. 
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CONCLUSIONS 


Introduction 


The  study  objective  was  to  develop  and  evaluate  the  practical  use- 
fulness of  a paper-and-pencil  simulation  approach  to  performance  testing 
in  a technical  training  environment.  Satisfying  the  underlying  assump- 
tions necessary  to  normal  test  instrument  reliability  and  validity  mea- 
sures was  waived  in  the  study  design  to  permit  the  demonstration  to  em- 
brace as  many  of  the  real-world  variables  as  possible.  Emphasis  in  the 
design  of  the  SPT  was  on  content  validity. 


The  major  study  variables  -.’hich  militated  against  demonstration  of 
predictive  validity  included: 

1.  Items  of  widely  varying  difficulty. 

2.  Tests  with  few  items  (N  = 3). 

3.  Subject  groups  of  vastly  differing  backgrounds 
(students  and  instructors). 

4.  Small  subject  groups  (Students  = 15,  Instructors  = 16). 

5.  Tests  on  two  different  pieces  of  equipment. 

6.  Counterbalanced  order  of  presentation  for  equipment, 
test  format,  and  item. 

In  spite  of  the  action  of  these  factors,  significant  predictive 
validity  was  shown.  Comparison  of  all  subjects'  JST  versus  SPT  per- 
formance measures  produced  positive  and  statistically  significant  cor- 
relations. The  correlation  coefficients  for  accuracy,  time  to  solution, 
and  steps  to  solution  are  +.384,  +.588,  and  +.356  respectively. 


The  following  section  reviews  the  summary  statistics  for  each  of 
these  measures  to  assess  similarity  of  performance.  SPT  content  valid- 
ity and  the  problem  difficulty  metric  are  also  examined  in  these  con- 
clusions. 
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Accuracy 


The  pooled  (all  subjects,  correct  versus  incorrect)  accuracy  cor- 
relation was  positive  and  significant;  however,  the  potential  magnitude 
of  the  correlation  was  suppressed  by  the  variance  in  performance.  This 
variance  is  contributed  by: 

1.  Problem  difficulty  differences. 

2.  Low  overall  accuracy  (37.7%). 

3.  Student  and  instructor  differences. 

4.  Oscillator  and  oscilloscope  problem  differences. 

Table  1 compares  the  accuracy  correlations  with  the  GK  gammas.  With 
the  exception  of  the  instructor  and  student  JST/SPT  correlations,  the 
gammas  closely  parallel  the  correlations.  If  one  accepts  the  gammas  as 
more  indicative  of  performance  due  to  their  associative  nature,  we  must 
conclude  that  performance,  accuracy-wise,  was  fairly  uniform  across  these 
categories . 

TABLE  1.  COMPARISON  OF  ACCURACY  METRICS 


JST/SPT 

Instructor 

JST/SPT 

Student 

JST/SPT 

Oscillator 

JST/SPT 

Scope 

JST/SPT 

r = +0.3L 

+0.63 

-0.04 

+0.29 

+0.48 

GK  gamma  » +0.38 

+0.31 

+0.44 

+0.31 

+0.44 

Time  Correlations 


The  time  differences  are  primarily  a function  of  the  order  of  pre- 
sentation rather  than  test  format.  The  increased  time  per  step  in  SPT 
tests  is  probably  a function  of  the  volume  of  SPT  materials  which  must  be 
used  to  solve  a problem.  This  limitation  may  have  forced  the  subjects  to 
consider  more  carefully  their  strategy  while  performing  the  SPTs.  The 
test  structure  may  also  have  eliminated  some  irrelevant  checks.  These 
conclusions  are  not  obvious  fVtvn  the  collected  data;  however,  many  sub- 
jects commented  that  the  symbol!.;  test  was  more  difficult  and  that  they 
learned  a lot  as  a result  of  the  testing. 

The  time  correlations  were  high,  positive,  and  significant,  and  lend 
support  to  the  claim  for  test  similarity  made  on  behalf  of  accuracy.  This 
argument  is  valid  since  the  tasks  performed  in  different  treatment  modes 
were  similar  in  approach  as  seen  through  the  protocol  analysis  results. 
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Problem  Difficulty 

Correlational  methods  did  not  produce  significant  agreement  between 
the  hypothesized  problem  difficulties  and  the  empirically  determined  re- 
sults. The  correlation  for  combined  JST  and  SPT  difficulties  compared 
to  the  hypothesized  ones  was  +.567  which  was  not  significant.  However, 
a t-test  performed  on  the  same  data  suggests  that  there  is  no  significant 
difference  between  the  means.  The  t-statistic  is  .886  which  is  not  sig- 
nificant at  the  .05  level.  Table  2 compares  the  hypothesized  and  empir- 
ical item  difficulties. 


TABLE  2.  HYPOTHESIZED  AND  EMPIRICAL  ITEM  DIFFICULTIES 


Difficulty  Factor 

Item 

SI 

S2 

S3 

01 

02 

03 

Hypothesized 

51.7 

77.4 

40.8 

64.7 

47.8 

48.2 

JST/SPT  Combined 

54.8 

90.3 

24.2 

51.6 

82.3 

71.0 

JST 

51.6 

96.8 

29.0 

51.6 

77.4 

77.4 

SPT 

58.1 

83.9 

19.4 

51.6 

87.1 

64.5 

The  major  differences 

occurred 

on  oscillator 

problems  2 and 

: 3. 

Reviewing  the  earlier  accuracy  data  reveals  that  both  subject  groups  ap- 
peared to  have  more  than  the  expected  difficulty  with  these  problems. 
This  finding  suggests  that  the  difference  is  equipment-related.  A major 
difference  between  the  oscillator  and  the  scope  is  the  extensive  use  of 
both  positive  and  negative  feedback  in  the  oscillator  circuitry.  There 
is  no  question  that  feedback  complicates  the  troubleshooting  task.  The 
unanswered  question  is  whether  the  subject  group  is  typical  of  the  gen- 
eral population  in  this  regard.  If  so,  the  problem  difficulty  algorithm 
is  insensitive  to  this  hardware  feature. 

While  these  results  indicate  that  the  algorithm  requires  some  fine- 
tuning,  the  algorithm  has  been  partially  proven.  The  importance  of  the 
algorithm  cannot  be  overstated.  In  future  tests  of  this  sort,  devel- 
opers will  be  limited  by  item  development  cost;  prohibited  from  build- 
ing a pool  of  items  to  establish  empirical  difficulty.  The  algorithm 
is  at  least  a yardstick  for  gauging  difficulty  which  can  permit  test 
developers  to  approximate  difficulty  and  select  roughly  equivalent  items 
for  reliability  and  validity  determinations. 
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Similarity  of  Performance 


Similar  performance  is  a measure  of  concordance  (e.g.,  what  percent- 
age of  the  time  were  the  subjects'  responses  for  an  item  the  same  between 
test  types:  either  both  correct,  or  both  incorrect).  Table  3 compares 
two  sets  of  concordant  data.  The  gammas  are  Goodman-Kruskal  indices  of 
concordance  which  have  the  same  range  and  interpretation  as  a correlation 
coefficient.  The  percent  matched  sets  are  determined  as  follows: 

% matched  sets  = £ (JST=SPT) 

£ Response  Sets 


TABLE  3. 

COMPARISON 

OF  CONCORDANT  ACCURACY 

RESULTS 

JST/SPT 

Instructor 

JST/SPT 

Student 

JST/SPT 

Oscillator 

JST/SPT 

Scope 

JST/SPT 

GK  gamma 

+0.38 

+0.31 

+0.44 

+0.31 

+0.44 

% matched 
sets 

68.8 

65.6 

72.0 

65.5 

74.0 

The  protocol  data  confirm  performance  similarity  in  terms  of  strat- 
egy. Organized  strategies  were  present  in  both  JST  and  SPT  solutions. 
The  protocol  results  produced  high,  positive,  and  significant  correla- 
tions (+0.96,  +0.93,  and  +0.75)  in  the  organized  approaches  categories. 

The  grand  means  for  time,  accuracy,  and  steps  to  completion  also 
suggest  little  overall  difference  between  JST  and  SPT  treatments.  These 
means  are  presented  in  Table  4. 

TABLE  4.  COMPARISON  OF  OUTPUT  MEASURES  BY  TREATMENT  TYPE 

Accuracy  Time  to  Complete  Steps  to  Complete 

JST  36.6%  29.7  5.9 

SPT  39.3%  29.2  4.5 

When  combined,  the  evidence  overwhelmingly  suggests  a high  degree 
of  similarity  between  job  sample  and  symbolic  performance. 


SPT  Content  Validity 


The  study  was  focused  on  troubleshooting  to  the  stage  level.  The 
subtasks  which  make  up  generic  troubleshooting  are: 

1.  Manipulating  equipment  system  state  to  discern 
the  character  or  probable  location  of  the  mal- 
function. 

2.  Measuring  operating  signals  anywhere  within  the 
equipment. 

3.  Selecting  from  normally  available  test  equipment 
to  measure  operating  signals. 

4.  Physically  locating  the  desired  point  of  test. 

5.  Choosing  a measurement  parameter. 

6.  Adjusting  test  equipment  to  measure  the  desired 
operating  parameter. 

7.  Reading  displays  and  factoring  range  and  function 
information  into  values. 

8.  Planning  or  modifying  troubleshooting  strategy 
based  on  test  results. 

9.  Using  equipment  technical  data  during  trouble- 
shooting. 

The  SPT  materials  used  in  this  study  addressed  the  component  sub- 
tasks in  generic  troubleshooting  by: 

1.  Providing  access  to  front  panel,  controls  to  allow 
changing  system  state. 

2.  Providing  access  to  front  panel  displays  in  response 
to  system  state  changes. 

3.  Providing  access  to  all  points  of  test. 

4.  Providing  access  to  normally  available  test 
equipment . 

5.  Permitting  subjects  the  choice  of  measurement 
parameters . 

6.  Providing  access  to  test  equipment  range  and  function 
controls  to  permit  selection  of  test  parameters. 
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7.  Providing  test  equipment  displays  to  permit  factoring 
of  range  and  function  information  into  values. 

8.  Designing  the  SPT  materials  to  permit  the  subjects 
to  follow  the  strategy  of  their  choice. 

9.  Providing  access  to  equipment  technical  data. 

If  one  accepts  that  the  list  of  generic  troubleshooting  subtasks 
is  complete  or  at  least  contains  the  most  important  elements  of  the 
troubleshooting  task,  then  one  must  conclude  that  the  SPT  content  is 
similar  to  on-equipment  troubleshooting  task  content. 

Upon  completion  of  the  testing,  each  subject  was  asked  to  give  the 
test  administrator  "feedback"  or  any  relevant  comments  on  either  test. 
While  the  comments  were  solicited,  they  were  provided  voluntarily.  Com- 
ments were  received  from  24  of  the  31  subjects.  Most  of  the  subjects 
attempted  to  provide  some  constructive  criticism  of  the  SPT.  These 
criticisms  have  been  mentioned  with  the  SPT  features  they  pertained  to 
elsewhere  in  this  report.  Approximately  40  percent  of  the  comments 
addressed  content  validity.  Typical  comments  were: 

"...  overall,  it  [the  SPT]  was  quite  different  from  the 
standard  theoretical  troubleshooting  [presently  given  in 
training]  and  I believe  It's  better.  ..."  (An  instructor) 

".  . .1  think  it  [the  SPT]  was  a reasonably  good  method 
and  realistic  too.  I think  it  might  be  an  effective  aid 
to  [learning]  actual  troubleshooting.  You  must  apply  the 
same  mental  logic  in  either  method.  ..."  (A  student) 

".  . . once  you  get  used  to  the  system,  it  [the  SPT]  seems 
pretty  good.  . . ."  (An  instructor) 

".  . . the  way  that  the  paper-pencil  [SPT]  was  presented 
was  easily  understood  and  the  roubleshooting  was  easier 
[than  JST]  to  do.  I really  enjoyed  this  class  and  feel 
that  I have  learned  more  [than  in  the  present  course].  ..." 

(A  student) 

".  . .my  personal  preference  is  the  on-equipment  test.  The 
paper  troubleshooting  test  is  good  though.  I think  it  would 
be  excellent,  especially  where  equipment  is  not  available 
for  training.  . . ."  (A  student) 

Both  from  the  analytic  point  of  view  and  from  the  subjects'  reac- 
tions to  the  SPT,  one  must  conclude  that  the  SPT  content  validity  is 
quite  high. 


Discussion 


The  comparative  measures  individually  or  combined  illustrate  a high 
degree  of  performance  similarity  regardless  of  test  format.  The  order 
of  presentation  data  show  that  some  learning  occurred  between  presenta- 
tions and  that  the  instructor  JST  parameters  were  relatively  unaffected 
by  either  another  exposure  or  the  symbolic  format. 

The  reliability  and  validity  of  the  technique  as  a performance  mea- 
surement and  prediction  tool  are  yet  to  be  demonstrated.  However,  re- 
finement and  use  of  the  problem  difficulty  algorithm  should  reduce  this 
to  a routine  effort. 

The  most  provocative  result  is  the  similarity  of  performance  re- 
gardless of  format.  It  seems  very  likels-  that  the  symbolic  materials 
are  a substitute  for  the  actual  equipment  and  the  troubleshooting  en- 
vironment, and  could  be  used  to  practice  troubleshooting  in  a technical 
training  situation.  From  the  subjects'  comments,  one  could  expect  a 
fair  degree  of  SPT  acceptance  from  the  students  and  instructors  also. 

The  symbolic  materials  for  this  study  were  produced  in  a paper- 
and-pencil  medium;  however,  it  is  a short  step  from  paper  and  pencil 
to  a computer  presentation.  An  interactive  computer  terminal  coupled 
with  a graphics  terminal  or  carousel  projector  would  be  required  to 
replace  the  paper-and-pencil  format.  Whole  class  demonstrations  would 
be  possible  if  a slide  projector  was  used  to  display  the  graphics. 

This  change  in  medium  would  also  satisfy  most  of  the  critical  comments 
from  the  subjects  on  the  volume  of  SPT  materials. 


In  addition,  the  computer  could  be  utilized  in  a self-instructional 
mode.  The  optimal  strategy  can  be  programmed  in  addition  to  the 
malfunction-specific  data.  The  computer  can  provide  the  student  with 
immediate  feedback  at  each  decision  point  in  the  problem-solving  pro- 
cess on  the  student's  choice  of:  points  of  test,  test  equipment,  and 
test  equipment  range  and  function. 
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SUGGESTIONS  FOR  FUTURE  RESEARCH 


Introduction 


The  basic  goal  underlying  the  development  of  a valid  SPT  for  trouble- 
shooters is  the  potential  savings  in  training  equipment  by  substituting 
paper-and-pencil  tests  for  actual  operating  equipment.  This  saving  can 
be  viewed  as  increased  opportunity  for  each  student  to  interact  with  the 
equipment.  The  approach  featured  in  this  study  could  be  used  to  improve 
the  quality  of  training. 


Validity  and  Reliability  Stud\ 


Before  implementing  the  SPT  process  in  Air  Force  technical  training, 
it  is  necessary  to  demonstrate  its  validity  clearly  and  unequivocally. 
Based  on  the  present  study,  there  are  several  features  we  recommend  be 
part  of  a large-scale  study. 

1.  The  prime  equipment  used  in  the  study  should  be  widely 
used  in  the  field  so  that  a larger  pool  of  experienced 
service  personnel  could  be  sampled.  We  recommend  using 
only  one  piece  of  equipment  and  increasing  the  number 
of  problems  on  it.  This  also  simplifies  test  adminis- 
tration logistics  and  repair. 

2.  The  test  items  should  remain  typical  and  of  uniform 
difficulty. 

3.  The  point  of  test  location  visuals  should  be  in  color. 

The  specific  points  of  test  called  out  should  always 
include  individual  circuit  board  connectors. 


Computer  Presented  SPT 


The  SPT  materials  contained  two  types  of  visual  materials:  the  sets 
of  equipment  photographs  and  drawings,  and  the  cross-reference  matrices. 
The  matrices  were  the  objectionable  and  dif f icult-to-use  portions  of  the 
SPT  materials,  according  to  the  subjects.  Much  of  this  objection  could 
be  eliminated  if  the  tables  were  programmed  into  a computer  and  the  sub- 
ject simply  responded  to  a computer  prompt  for  each  decision  input. 
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The  point  of  test  location  photographs  would  provide  more  visual 
information  in  color;  they  were  black  and  white  in  this  study.  Shriver 
and  Foley  (1974)  suggest  the  use  of  a random-access  slide  projector  for 
visuals;  within  our  study  there  were  no  advantages  to  using  a projector. 
However,  a slide  projector  with  color  slides  for  point  of  test  and  range 
and  function  information  controlled  by  the  computer  would  seem  to  be  a 
natural  combination. 

The  validity  and  reliability  study  should  be  run  using  a computer 
presented  SPT.  The  visuals  should  be  computer  controlled  color  slides. 
The  computer  protocol  should  follow  the  decision  sequence  discussed 
earlier  in  this  report. 

The  results  should  then  be  carefully  compared  to  the  validation 
study  results.  An  important  feature  to  study  would  be  user  acceptance 
of  paper-and-pencil  presentation  versus  computer  presentation. 


SPT  Troubleshooting  Practice  Compared  with 
Conventional  On-Equipment  Troubleshooting  Practice 


The  data  reported  in  results  and  conclusions  suggest  that  the  SPT 
materials  lend  themselves  to  use  in  the  troubleshooting  practice  just 
as  effectively  as  the  actual  equipment.  If  symbolic  materials  were 
used  in  conjunction  with  the  protocol  records  and  protocol  maps,  an 
entire  class  could  practice  troubleshooting  without  any  operational 
equipment.  Subjects  could  self-score  the  protocol  maps  to  compare 
their  strategy  with  the  optimal  solution.  The  instructor  would  be  re- 
quired only  to  explain  and  discuss  differences  of  approach  that  were 
unclear  to  the  student. 

Experimental  and  control  classes  should  be  trained  in  troubleshoot- 
ing. The  control  class  should  be  trained  normally.  The  experimental 
class  should  use  SPT  materials  with  protocol  map  feedback  for  trouble- 
shooting practice.  Their  performance  should  then  be  compared  on  a cri- 
terion JST  and  against  the  control  class  JST  results. 


Problem  Difficulty  Algorithm 


As  additional  studies  are  conducted  and  additional  materials  pre- 
pared, the  problem  difficulty  ratings  should  be  calculated  and  the  re- 
sults compared  as  described  in  this  report.  Special  attention  should 
be  given  to  the  matter  of  algorithm  sensitivity  to  feedback  circuits. 
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The  algorithm  has  a multitude  of  other  applications  such  as  main- 
tainability evaluations  and  equipment  evaluation  in  the  context  of 
training  curriculum  development. 


SPT  Method  Applicability  to  Other  Levels  of  Troubleshooting 


The  SPT  method  developed  on  this  project  can  be  used  for  any  level 
of  troubleshooting  penetration.  The  items  developed  and  tested  on  this 
project  could  be  used  to  test  or  practice  component  level  troubleshooting 
by  simply  having  the  subject  answer  with  a component  designator  rather 
than  a stage  identification.  It  is  possible  to  build  SPT  materials  such 
as  these  to  cover  black  box,  stage  or  chassis,  and  component  level  repair 
for  the  cost  of  item  development  for  troubleshooting  to  the  stage  level. 
This  would  seem  to  be  a desirable  feature,  since  tri  ubleshooting  could 
be  taught,  practiced,  and  tested  as  it  is  performed  in  the  field. 

It  would  be  desirable  to  rerun  this  study  for  component  level 
troubleshooting  to  demonstrate  the  applicability  of  the  concept  at  this 
level.  Extreme  caution  should  be  exercised  when  selecting  subjects  since 
the  level  of  difficulty  will  increase  and  the  proposed  study  will  suffer 
from  low  overall  accuracy  if  competent  subjects  are  not  used. 
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SYMBOLIC  PERFORMANCE  TEST 
SAMPLE  PROBLEM 
SCHEMATIC  (TECHNICAL  DATA) 
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SAMPLE  PROBLEM  1 
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SAMPLE  PROBLEM  2 


VOLTMETER  CONVERSION  TABLE:  OHMS 
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